[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:38 dist-test-slave-zqlt krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:38 dist-test-slave-zqlt krb5kdc[399](info): set up 2 sockets
Sep 28 17:51:38 dist-test-slave-zqlt krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 17:51:39.292510 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:39.411005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:39.491950 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:39.492280 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:39.492322 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:39.493511 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:39.493587 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:39.493613 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:39.493635 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:39.493664 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:39.495144 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:39.495425 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:39.495673 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:39.500483 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.500478 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.500567 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:39.500608 412 server_base.cc:1034] running on GCE node
I20260928 17:51:39.501003 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:39.501335 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:39.502504 412 hybrid_clock.cc:648] HybridClock initialized: now 1790617899502501 us; error 32 us; skew 500 ppm
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:39.505023 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:39.505482 412 webserver.cc:460] Webserver started at http://127.0.98.62:34845/ using document root <none> and password file <none>
I20260928 17:51:39.505720 412 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:39.505834 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:39.506031 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:39.507090 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "716180dbb38e47a8bf68ffdea0f033c0"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "5efdbf138af04520c4b36db80b9d8347"
server_key_iv: "1459c509ba82b77870164dbb349d1c38"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.507452 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "716180dbb38e47a8bf68ffdea0f033c0"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "5efdbf138af04520c4b36db80b9d8347"
server_key_iv: "1459c509ba82b77870164dbb349d1c38"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.509249 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:51:39.510001 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.510260 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:39.510350 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "716180dbb38e47a8bf68ffdea0f033c0"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "5efdbf138af04520c4b36db80b9d8347"
server_key_iv: "1459c509ba82b77870164dbb349d1c38"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.510429 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:39.529596 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:39.531930 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:39.532142 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:39.535996 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35559
I20260928 17:51:39.536031 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35559 every 8 connection(s)
I20260928 17:51:39.536425 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:39.537266 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:39.539481 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0: Bootstrap starting.
I20260928 17:51:39.540059 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:39.540367 480 log.cc:824] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:39.541028 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0: No bootstrap required, opened a new log
I20260928 17:51:39.542567 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260928 17:51:39.542713 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:39.542737 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716180dbb38e47a8bf68ffdea0f033c0, State: Initialized, Role: FOLLOWER
I20260928 17:51:39.542899 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260928 17:51:39.542956 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:39.543013 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:39.543063 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:39.543578 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260928 17:51:39.543692 480 leader_election.cc:302] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 716180dbb38e47a8bf68ffdea0f033c0; no voters:
I20260928 17:51:39.543869 480 leader_election.cc:288] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:39.543905 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:39.544052 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [term 1 LEADER]: Becoming Leader. State: Replica: 716180dbb38e47a8bf68ffdea0f033c0, State: Running, Role: LEADER
I20260928 17:51:39.544137 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:39.544229 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260928 17:51:39.544646 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 716180dbb38e47a8bf68ffdea0f033c0. Latest consensus state: current_term: 1 leader_uuid: "716180dbb38e47a8bf68ffdea0f033c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260928 17:51:39.544719 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:39.544780 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "716180dbb38e47a8bf68ffdea0f033c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "716180dbb38e47a8bf68ffdea0f033c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260928 17:51:39.544857 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:39.545148 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:39.545727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 412
I20260928 17:51:39.545780 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:39.545843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:39.546563 392 external_mini_cluster.cc:1428] Setting key 74d79539a0da6f0aee99479221b7a96d
I20260928 17:51:39.547358 492 catalog_manager.cc:1347] Generated new cluster ID: 27df090fb66f4271b02c9acc0133479c
I20260928 17:51:39.547959 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:39.563910 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:39.547826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.98.62:35559)
0928 17:51:39.547983 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:39.547992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:39.548026 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:51:39.549946 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:39.549953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:39.550009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:39.550154 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:39.552062 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.553827 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.555181 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.555762 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.555891 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:39.559973 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:39.559977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:39.559977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:39.559987 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:39.561984 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.562464 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.562466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.562467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.562563 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.562844 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.562844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.562845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.563135 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:39.563286 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:39.563565 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 17:51:39.563703 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 17:51:39.566061 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:39.566761 492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:39.583192 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 716180dbb38e47a8bf68ffdea0f033c0: Generated new TSK 0
I20260928 17:51:39.583364 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:39.615105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:39.697546 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:39.697785 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:39.697824 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:39.698953 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:39.699021 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:39.699067 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:39.700543 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:39.700814 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:39.701041 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:39.705901 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.705917 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.705991 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:39.706147 506 server_base.cc:1034] running on GCE node
I20260928 17:51:39.706488 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:39.706754 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:39.707911 506 hybrid_clock.cc:648] HybridClock initialized: now 1790617899707889 us; error 36 us; skew 500 ppm
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:39.710325 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:39.710736 506 webserver.cc:460] Webserver started at http://127.0.98.1:37613/ using document root <none> and password file <none>
I20260928 17:51:39.710965 506 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:39.711030 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:39.711179 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:39.712283 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d67e38aeb104786afa5abcbb164be31"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "68e525ad660c2e8ac7a4bc526bd4ab8d"
server_key_iv: "ec7f786ad4a462d4874c2c4ae7f11f58"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.712656 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d67e38aeb104786afa5abcbb164be31"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "68e525ad660c2e8ac7a4bc526bd4ab8d"
server_key_iv: "ec7f786ad4a462d4874c2c4ae7f11f58"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.713997 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:39.714694 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.714857 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:39.714955 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "6d67e38aeb104786afa5abcbb164be31"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "68e525ad660c2e8ac7a4bc526bd4ab8d"
server_key_iv: "ec7f786ad4a462d4874c2c4ae7f11f58"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.715026 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:39.738435 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:39.740711 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:39.740875 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:39.741117 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:39.741429 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:39.741482 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.741541 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:39.741585 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.747699 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34517
I20260928 17:51:39.747774 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34517 every 8 connection(s)
I20260928 17:51:39.748134 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:51:39.750270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 506
I20260928 17:51:39.750375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:39.750519 392 external_mini_cluster.cc:1428] Setting key 42cf0f874c2604a0ed8e967841fe81a7
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:39.758769 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:39.748939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36753 (local address 127.0.98.62:35559)
0928 17:51:39.749007 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:39.749010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:39.749230 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:51:39.749697 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:39.749698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:39.749707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:39.749766 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:39.750864 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.751465 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.752835 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.753035 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.753095 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:39.755621 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:39.755623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:39.755623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:39.755632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:39.757383 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.757773 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.757774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.757775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.757833 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.758072 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.758073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.758073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.758223 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:39.758301 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:39.758550 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 17:51:39.758666 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:51:39.758796 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:39.748927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.1:36753)
0928 17:51:39.749094 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:39.749138 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:39.749611 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:39.749821 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:39.749825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:39.750472 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:39.750743 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:39.750749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.751603 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.751604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:39.752739 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:39.752743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.753121 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.753123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:39.753273 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:39.753891 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:39.753899 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:39.755516 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:39.757472 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:39.757477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:39.757479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:39.757698 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:39.757901 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:39.757902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:39.757902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:39.757994 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:39.758292 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:39.758296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:39.758411 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 17:51:39.758494 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:51:39.758626 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:51:39.759320 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260928 17:51:39.759425 635 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:39.759637 635 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260928 17:51:39.760197 444 ts_manager.cc:194] Registered new tserver with Master: 6d67e38aeb104786afa5abcbb164be31 (127.0.98.1:34517)
I20260928 17:51:39.761255 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:39.776916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:39.857295 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:39.857544 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:39.857584 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:39.858772 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:39.858844 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:39.858891 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:39.860399 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:39.860678 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:39.860905 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:39.866292 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.866302 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:39.866307 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:39.866406 640 server_base.cc:1034] running on GCE node
I20260928 17:51:39.866700 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:39.866942 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:39.868121 640 hybrid_clock.cc:648] HybridClock initialized: now 1790617899868085 us; error 48 us; skew 500 ppm
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:39.870639 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:39.871106 640 webserver.cc:460] Webserver started at http://127.0.98.2:39831/ using document root <none> and password file <none>
I20260928 17:51:39.871290 640 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:39.871330 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:39.871423 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:39.872517 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b5476c1ac3d4c8bbb63991d80e027e8"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "8094a75537226305848ec8037b7fdf1c"
server_key_iv: "6508abf12380a3ea96e2e425ae5e7ad5"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.872829 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b5476c1ac3d4c8bbb63991d80e027e8"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "8094a75537226305848ec8037b7fdf1c"
server_key_iv: "6508abf12380a3ea96e2e425ae5e7ad5"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.874102 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:39.874805 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.875002 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:39.875092 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "1b5476c1ac3d4c8bbb63991d80e027e8"
format_stamp: "Formatted at 2026-09-28 17:51:39 on dist-test-slave-zqlt"
server_key: "8094a75537226305848ec8037b7fdf1c"
server_key_iv: "6508abf12380a3ea96e2e425ae5e7ad5"
server_key_version: "encryptionkey@0"
I20260928 17:51:39.875151 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:39.884253 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:39.886194 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:39.886313 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:39.886493 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:39.886838 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:39.886873 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.886919 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:39.886946 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:39.893496 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45959
I20260928 17:51:39.893532 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45959 every 8 connection(s)
I20260928 17:51:39.893870 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:51:39 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:39.901551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 640
I20260928 17:51:39.901634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:39.901830 392 external_mini_cluster.cc:1428] Setting key aabe8d7f1d08492faea4e2295155f536
I20260928 17:51:39.904304 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:39.894666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54185 (local address 127.0.98.62:35559)
0928 17:51:39.894740 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:39.894742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:39.894950 (+ 208us) server_negotiation.cc:407] Connection header received
0928 17:51:39.895443 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:39.895444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:39.895454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:39.895509 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:39.896493 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.896989 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.897956 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.898145 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.898203 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:39.900365 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:39.900367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:39.900367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:39.900374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:39.901587 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.902091 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.902092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.902093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.902158 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:39.902432 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:39.902433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:39.902433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:39.902587 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:39.902654 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:39.904072 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0928 17:51:39.904184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:51:39.904386 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:39.894653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.2:54185)
0928 17:51:39.894823 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:39.894860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:39.895359 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:39.895594 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:39.895598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:39.896208 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:39.896408 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:39.896413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.897076 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.897076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:39.897877 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:39.897878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:39.898222 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:39.898223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:39.898329 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:39.898811 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:39.898819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:39.900246 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:39.901706 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:39.901712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:39.901715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:39.902020 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:39.902228 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:39.902229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:39.902229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:39.902355 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:39.903795 (+ 1440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:39.903801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:39.903923 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 17:51:39.904018 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 17:51:39.904200 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260928 17:51:39.905279 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260928 17:51:39.905371 769 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:39.905553 769 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260928 17:51:39.905920 444 ts_manager.cc:194] Registered new tserver with Master: 1b5476c1ac3d4c8bbb63991d80e027e8 (127.0.98.2:45959)
I20260928 17:51:39.906306 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54185
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:39.926573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:40.008358 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:40.008575 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:40.008600 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:40.009773 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:40.009819 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:40.009845 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:40.011356 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:40.011603 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:40.011828 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:40.016853 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:40.016861 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:40.016861 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:40.017050 774 server_base.cc:1034] running on GCE node
I20260928 17:51:40.017325 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:40.017594 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:40.018766 774 hybrid_clock.cc:648] HybridClock initialized: now 1790617900018753 us; error 42 us; skew 500 ppm
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:40.021303 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:40.021754 774 webserver.cc:460] Webserver started at http://127.0.98.3:35029/ using document root <none> and password file <none>
I20260928 17:51:40.021935 774 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:40.021974 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:40.022084 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:40.023085 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "fddc99938a134a8182f48e9072da3a3d"
format_stamp: "Formatted at 2026-09-28 17:51:40 on dist-test-slave-zqlt"
server_key: "ee5141ae582bd549f118ab193005275c"
server_key_iv: "21973ba080faa4bd6140104318fd3a98"
server_key_version: "encryptionkey@0"
I20260928 17:51:40.023401 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fddc99938a134a8182f48e9072da3a3d"
format_stamp: "Formatted at 2026-09-28 17:51:40 on dist-test-slave-zqlt"
server_key: "ee5141ae582bd549f118ab193005275c"
server_key_iv: "21973ba080faa4bd6140104318fd3a98"
server_key_version: "encryptionkey@0"
I20260928 17:51:40.024950 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:51:40.025732 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:40.025938 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:40.026038 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "fddc99938a134a8182f48e9072da3a3d"
format_stamp: "Formatted at 2026-09-28 17:51:40 on dist-test-slave-zqlt"
server_key: "ee5141ae582bd549f118ab193005275c"
server_key_iv: "21973ba080faa4bd6140104318fd3a98"
server_key_version: "encryptionkey@0"
I20260928 17:51:40.026098 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:40.035518 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:40.037590 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:40.037714 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:40.037910 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:40.038177 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:40.038208 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:40.038230 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:40.038242 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:40.045107 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37969
I20260928 17:51:40.045249 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37969 every 8 connection(s)
I20260928 17:51:40.045583 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:40.051791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 774
I20260928 17:51:40.051878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:40.052078 392 external_mini_cluster.cc:1428] Setting key c47b6b847201ff63db3281331a2f0d76
I20260928 17:51:40.055321 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:40.046389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49731 (local address 127.0.98.62:35559)
0928 17:51:40.046451 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:40.046454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:40.046727 (+ 273us) server_negotiation.cc:407] Connection header received
0928 17:51:40.047197 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:40.047198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:40.047208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:40.047272 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:40.048359 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.048864 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.049895 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.050087 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.050158 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:40.052581 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:40.052583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:40.052583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:40.052592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:40.054075 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.054442 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.054443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.054443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.054499 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.054735 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.054736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.054737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.054850 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:40.054907 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:40.055175 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 17:51:40.055257 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:51:40.055405 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:40.046376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.3:49731)
0928 17:51:40.046553 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:40.046598 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:40.047106 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:40.047341 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:40.047344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:40.047959 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:40.048262 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:40.048269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.048948 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.048949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:40.049807 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:40.049809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.050175 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.050176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:40.050278 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:40.050878 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:40.050888 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:40.052443 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:40.054148 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:40.054155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:40.054156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:40.054374 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:40.054567 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:40.054567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:40.054568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:40.054675 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:40.054908 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:40.054912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:40.055028 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:51:40.055111 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:51:40.055237 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260928 17:51:40.055917 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260928 17:51:40.056005 903 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:40.056227 903 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260928 17:51:40.056658 444 ts_manager.cc:194] Registered new tserver with Master: fddc99938a134a8182f48e9072da3a3d (127.0.98.3:37969)
I20260928 17:51:40.057001 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49731
I20260928 17:51:40.064568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:40.088440 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:40.081059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.0.98.1:34517)
0928 17:51:40.081238 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:40.081242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:40.081253 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:51:40.081279 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:40.081282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:40.081295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:40.081362 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:40.081879 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.082565 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.083419 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.083760 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.083816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:40.085939 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:40.085947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:40.085947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:40.085954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:40.087336 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.087643 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.087646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.087646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.087708 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.087864 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.087865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.087865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.087986 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:40.088162 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:40.088225 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:51:40.088287 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
W20260928 17:51:40.088821 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51834
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:40.110580 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:40.103810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.98.1:34517)
0928 17:51:40.103884 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:40.103886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:40.103915 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:51:40.103988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:40.103989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:40.104000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:40.104063 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:40.104630 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.105113 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.105791 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:40.105979 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:40.106037 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:40.108309 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:40.108311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:40.108311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:40.108319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:40.109714 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.109999 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.110000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.110000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.110058 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:40.110197 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:40.110197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:40.110197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:40.110275 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:40.110358 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:40.110419 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 17:51:40.110476 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:51:40.111148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 506
I20260928 17:51:40.116716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 640
I20260928 17:51:40.122202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 774
I20260928 17:51:40.126394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 412
2026-09-28T17:51:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1887 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[917](info): set up 2 sockets
Sep 28 17:51:40 dist-test-slave-zqlt krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 17:51:41.160066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:41.278762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:41.358929 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:41.359205 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:41.359249 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:41.360613 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:41.360689 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:41.360716 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:41.360746 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:41.360780 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:41.362480 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:41.362787 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:41.363034 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:41.368500 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:41.368523 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:41.368501 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:41.368536 930 server_base.cc:1034] running on GCE node
I20260928 17:51:41.369056 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:41.369346 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:41.370496 930 hybrid_clock.cc:648] HybridClock initialized: now 1790617901370485 us; error 28 us; skew 500 ppm
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:41.373183 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:41.373627 930 webserver.cc:460] Webserver started at http://127.0.98.62:44989/ using document root <none> and password file <none>
I20260928 17:51:41.373904 930 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:41.373976 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:41.374125 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:41.375212 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "8cb6a89a428543a2a509a0497efd601c"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "e742b95342284d4f563e7321ae9fc7ec"
server_key_iv: "c5d346b4fcb65d20fc1349573f24e54d"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.375574 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cb6a89a428543a2a509a0497efd601c"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "e742b95342284d4f563e7321ae9fc7ec"
server_key_iv: "c5d346b4fcb65d20fc1349573f24e54d"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.377025 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:41.377796 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.377969 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:41.378062 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "8cb6a89a428543a2a509a0497efd601c"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "e742b95342284d4f563e7321ae9fc7ec"
server_key_iv: "c5d346b4fcb65d20fc1349573f24e54d"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.378160 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:41.395623 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:41.397794 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:41.397979 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:41.401882 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42199
I20260928 17:51:41.401916 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42199 every 8 connection(s)
I20260928 17:51:41.402297 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:41.403174 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:41.403406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 930
I20260928 17:51:41.403503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:41.403663 392 external_mini_cluster.cc:1428] Setting key cd689379680267657c14590b84b5edc6
I20260928 17:51:41.406760 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c: Bootstrap starting.
I20260928 17:51:41.407454 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:41.407749 998 log.cc:824] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:41.408595 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c: No bootstrap required, opened a new log
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:41.410740 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } }
I20260928 17:51:41.410933 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:41.410979 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb6a89a428543a2a509a0497efd601c, State: Initialized, Role: FOLLOWER
I20260928 17:51:41.411114 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } }
I20260928 17:51:41.411290 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:41.411386 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:41.411445 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:41.412323 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } }
I20260928 17:51:41.412432 998 leader_election.cc:302] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cb6a89a428543a2a509a0497efd601c; no voters:
I20260928 17:51:41.412660 998 leader_election.cc:288] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:41.412786 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:41.412930 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [term 1 LEADER]: Becoming Leader. State: Replica: 8cb6a89a428543a2a509a0497efd601c, State: Running, Role: LEADER
I20260928 17:51:41.412951 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:41.413030 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } }
I20260928 17:51:41.413360 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cb6a89a428543a2a509a0497efd601c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } } }
I20260928 17:51:41.413457 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:41.413615 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cb6a89a428543a2a509a0497efd601c. Latest consensus state: current_term: 1 leader_uuid: "8cb6a89a428543a2a509a0497efd601c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cb6a89a428543a2a509a0497efd601c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42199 } } }
I20260928 17:51:41.413688 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:41.413802 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:41.414336 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:41.415946 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:41.404216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.98.62:42199)
0928 17:51:41.404414 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:41.404425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:41.404458 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:51:41.404945 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:41.404955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:41.405024 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:41.405182 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:41.405750 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.407149 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.408011 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.408679 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.408795 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:41.411409 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:41.411413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:41.411413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:41.411427 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:41.413768 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.414204 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.414208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.414209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.414322 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.414555 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.414556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.414556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.414889 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:41.415004 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:41.415567 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 17:51:41.415730 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260928 17:51:41.420733 1010 catalog_manager.cc:1347] Generated new cluster ID: cfa592678ae743dcb96f6b53ad3659c2
I20260928 17:51:41.420828 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:51:41.438876 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:41.439469 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:41.443115 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cb6a89a428543a2a509a0497efd601c: Generated new TSK 0
I20260928 17:51:41.443315 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:41.477012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:41.557472 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:41.557713 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:41.557758 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:41.558930 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:41.559000 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:41.559046 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:41.560490 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:41.560760 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:41.560985 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:41.566354 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:41.566360 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:41.566479 1024 server_base.cc:1034] running on GCE node
W20260928 17:51:41.566406 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:41.566758 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:41.567034 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:41.568167 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790617901568130 us; error 35 us; skew 500 ppm
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:41.570639 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:41.571062 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45519/ using document root <none> and password file <none>
I20260928 17:51:41.571306 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:41.571374 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:41.571544 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:41.572671 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c2f447d53a54c3f9c7adde292198e0f"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "bcc6af47abaa528883634c0724ef5b34"
server_key_iv: "aba562cc02f3b4bd7800b7423335c375"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.573025 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c2f447d53a54c3f9c7adde292198e0f"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "bcc6af47abaa528883634c0724ef5b34"
server_key_iv: "aba562cc02f3b4bd7800b7423335c375"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.574317 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:41.574982 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.575141 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:41.575243 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "2c2f447d53a54c3f9c7adde292198e0f"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "bcc6af47abaa528883634c0724ef5b34"
server_key_iv: "aba562cc02f3b4bd7800b7423335c375"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.575323 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:41.586081 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:41.587952 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:41.588171 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:41.588409 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:41.588711 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:41.588769 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.588804 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:41.588830 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.594698 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35941
I20260928 17:51:41.594769 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35941 every 8 connection(s)
I20260928 17:51:41.595067 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:41.602041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1024
I20260928 17:51:41.602114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:41.602304 392 external_mini_cluster.cc:1428] Setting key 96ec856d818078a2a949662d0ec5711e
I20260928 17:51:41.604544 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:41.595793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53873 (local address 127.0.98.62:42199)
0928 17:51:41.595865 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:41.595867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:41.596139 (+ 272us) server_negotiation.cc:407] Connection header received
0928 17:51:41.596620 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:41.596621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:41.596631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:41.596691 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:41.597622 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.598172 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.599128 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.599332 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.599400 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:41.601581 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:41.601582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:41.601582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:41.601589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:41.603143 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.603559 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.603561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.603561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.603619 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.603888 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.603889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.603889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.604003 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:41.604079 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:41.604371 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 17:51:41.604456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:51:41.604753 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:41.595774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42199 (local address 127.0.98.1:53873)
0928 17:51:41.595934 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:41.595975 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:41.596524 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:41.596749 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:41.596752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:41.597320 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:41.597517 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:41.597522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.598255 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.598255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:41.599042 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:41.599043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.599408 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.599409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:41.599507 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:41.599992 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:41.600000 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:41.601494 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:41.603258 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:41.603265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:41.603267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:41.603482 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:41.603701 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:41.603702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:41.603702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:41.603817 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:41.604071 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:41.604077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:41.604239 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 17:51:41.604350 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 17:51:41.604505 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 17:51:41.605248 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42199
I20260928 17:51:41.605331 1153 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:41.605520 1153 heartbeater.cc:511] Master 127.0.98.62:42199 requested a full tablet report, sending...
I20260928 17:51:41.606019 962 ts_manager.cc:194] Registered new tserver with Master: 2c2f447d53a54c3f9c7adde292198e0f (127.0.98.1:35941)
I20260928 17:51:41.606671 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53873
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:41.627233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:41.712739 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:41.712950 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:41.712975 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:41.714366 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:41.714421 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:41.714459 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:41.716063 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:41.716341 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:41.716590 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:41.721778 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:41.721778 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:41.721766 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:41.721946 1158 server_base.cc:1034] running on GCE node
I20260928 17:51:41.722159 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:41.722398 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:41.723544 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790617901723532 us; error 41 us; skew 500 ppm
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:41.726272 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:41.726752 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35623/ using document root <none> and password file <none>
I20260928 17:51:41.727029 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:41.727082 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:41.727221 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:41.728413 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "de349f915ef2402a8bd721be9aeebded"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "c3fc2cd3e3aef8dd5e411fa4fb556d58"
server_key_iv: "a7d7362ea015ea4782120afa4c40437a"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.728801 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de349f915ef2402a8bd721be9aeebded"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "c3fc2cd3e3aef8dd5e411fa4fb556d58"
server_key_iv: "a7d7362ea015ea4782120afa4c40437a"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.730176 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:41.730942 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.731140 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:41.731245 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "de349f915ef2402a8bd721be9aeebded"
format_stamp: "Formatted at 2026-09-28 17:51:41 on dist-test-slave-zqlt"
server_key: "c3fc2cd3e3aef8dd5e411fa4fb556d58"
server_key_iv: "a7d7362ea015ea4782120afa4c40437a"
server_key_version: "encryptionkey@0"
I20260928 17:51:41.731328 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:41.745888 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:41.748463 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:41.748636 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:41.748885 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:41.749217 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:41.749276 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.749325 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:41.749352 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:41.755692 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45443
I20260928 17:51:41.755771 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45443 every 8 connection(s)
I20260928 17:51:41.756170 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:41.762171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1158
I20260928 17:51:41.762269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:41.762428 392 external_mini_cluster.cc:1428] Setting key e9d606f9c984d2f7746b358ed17f4772
Sep 28 17:51:41 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:41.766557 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:41.756919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38391 (local address 127.0.98.62:42199)
0928 17:51:41.756983 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:41.756985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:41.757185 (+ 200us) server_negotiation.cc:407] Connection header received
0928 17:51:41.757731 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:41.757732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:41.757743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:41.757803 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:41.758871 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.759788 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.760803 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.760990 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.761043 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:41.763381 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:41.763383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:41.763383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:41.763391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:41.765022 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.765514 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.765516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.765516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.765578 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:41.765874 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:41.765875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:41.765876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:41.766025 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:41.766111 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:41.766422 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 17:51:41.766490 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:51:41.767032 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:41.756873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42199 (local address 127.0.98.2:38391)
0928 17:51:41.757049 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:41.757093 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:41.757614 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:41.757865 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:41.757868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:41.758489 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:41.758742 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:41.758748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.759893 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.759894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:41.760704 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:41.760705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:41.761059 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:41.761059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:41.761159 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:41.761608 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:41.761616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:41.763289 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:41.765102 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:41.765109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:41.765111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:41.765432 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:41.765648 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:41.765648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:41.765649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:41.765801 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:41.766102 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:41.766109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:41.766283 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 17:51:41.766400 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 17:51:41.766584 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 17:51:41.767611 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42199
I20260928 17:51:41.767737 1287 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:41.768013 1287 heartbeater.cc:511] Master 127.0.98.62:42199 requested a full tablet report, sending...
I20260928 17:51:41.768601 962 ts_manager.cc:194] Registered new tserver with Master: de349f915ef2402a8bd721be9aeebded (127.0.98.2:45443)
I20260928 17:51:41.769004 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38391
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 17:51:42.402204 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.639s user 0.000s sys 0.003s
I20260928 17:51:42.402594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:42.489502 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:42.489717 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:42.489741 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:42.490921 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:42.490963 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:42.491021 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:42.492528 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42199
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:42.492771 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:42.493000 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:42.498045 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:42.498137 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:42.498035 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:42.499003 1292 server_base.cc:1034] running on GCE node
I20260928 17:51:42.499217 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:42.499460 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:42.500604 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790617902500593 us; error 32 us; skew 500 ppm
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:42.506331 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:42.506767 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45607/ using document root <none> and password file <none>
I20260928 17:51:42.507061 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:42.507138 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:42.507280 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:42.508577 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9480f19fc374f63986dd546fbf9ed1e"
format_stamp: "Formatted at 2026-09-28 17:51:42 on dist-test-slave-zqlt"
server_key: "987015f08038c416c658e91662d8afcb"
server_key_iv: "ef067ed5cb113c31e409d9d6394e2be0"
server_key_version: "encryptionkey@0"
I20260928 17:51:42.508927 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9480f19fc374f63986dd546fbf9ed1e"
format_stamp: "Formatted at 2026-09-28 17:51:42 on dist-test-slave-zqlt"
server_key: "987015f08038c416c658e91662d8afcb"
server_key_iv: "ef067ed5cb113c31e409d9d6394e2be0"
server_key_version: "encryptionkey@0"
I20260928 17:51:42.510241 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:42.510977 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:42.511157 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:42.511252 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "f9480f19fc374f63986dd546fbf9ed1e"
format_stamp: "Formatted at 2026-09-28 17:51:42 on dist-test-slave-zqlt"
server_key: "987015f08038c416c658e91662d8afcb"
server_key_iv: "ef067ed5cb113c31e409d9d6394e2be0"
server_key_version: "encryptionkey@0"
I20260928 17:51:42.511336 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:42.528726 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:42.530505 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:42.530671 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:42.530890 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:42.531224 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:42.531278 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:42.531312 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:42.531339 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:42.537020 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42091
I20260928 17:51:42.537094 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42091 every 8 connection(s)
I20260928 17:51:42.537412 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:42.546049 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.538196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35589 (local address 127.0.98.62:42199)
0928 17:51:42.538302 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.538304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.538473 (+ 169us) server_negotiation.cc:407] Connection header received
0928 17:51:42.538928 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.538929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.538941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.539002 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:42.539879 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.540429 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.541367 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.541570 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.541628 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.543595 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:42.543596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:42.543596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:42.543603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:42.544913 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.545280 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.545281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.545281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.545334 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.545535 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.545535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.545536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.545645 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:42.545698 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.545917 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.545980 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260928 17:51:42.546283 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.538168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42199 (local address 127.0.98.3:35589)
0928 17:51:42.538356 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.538388 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.538848 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.539052 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.539055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.539597 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:42.539778 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.539782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.540508 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.540509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.541289 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.541291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.541643 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.541644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.541740 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.542244 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:42.542251 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:42.543514 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:42.544989 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:42.544994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:42.544996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:42.545206 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:42.545390 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:42.545391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:42.545391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:42.545481 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:42.545701 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:42.545705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:42.545810 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.545890 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.546008 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260928 17:51:42.546815 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42199
I20260928 17:51:42.546911 1421 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:42.547079 1421 heartbeater.cc:511] Master 127.0.98.62:42199 requested a full tablet report, sending...
I20260928 17:51:42.547454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1292
I20260928 17:51:42.547488 962 ts_manager.cc:194] Registered new tserver with Master: f9480f19fc374f63986dd546fbf9ed1e (127.0.98.3:42091)
I20260928 17:51:42.547551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:42.547734 392 external_mini_cluster.cc:1428] Setting key b25a3fdaaa12ee3cec72c33c48f285e1
I20260928 17:51:42.547945 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35589
I20260928 17:51:42.548956 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:51:42.556222 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.550517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.0.98.62:42199)
0928 17:51:42.550582 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.550584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.550646 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:51:42.550697 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.550698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.550706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.550759 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:42.551271 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.551728 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.552492 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.552690 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.552747 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.553911 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:42.553912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:42.553912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:42.553917 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:42.555194 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.555515 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.555516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.555516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.555566 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.555729 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.555730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.555730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.555858 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:42.555936 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.556020 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.556083 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:51:42.558526 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:42.560191 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:42.566437 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.562656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.98.1:35941)
0928 17:51:42.562767 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.562772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.562937 (+ 165us) server_negotiation.cc:407] Connection header received
0928 17:51:42.562980 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.562982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.563000 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.563089 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:42.563191 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564070 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.565436 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566025 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566111 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566170 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.566315 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.566359 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260928 17:51:42.566427 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.562567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35941 (local address 127.0.0.1:34218)
0928 17:51:42.562795 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.562806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.562865 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.563085 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.563086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.563088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:42.563134 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.563135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564174 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.564175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.565340 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.565342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566109 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.566184 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566197 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.566297 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.566350 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260928 17:51:42.566849 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.562493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45443 (local address 127.0.0.1:53024)
0928 17:51:42.562733 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.562774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.562846 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.563047 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.563050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.563052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:42.563120 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.563122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564603 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.564604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.565639 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.565641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566589 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.566654 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566676 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.566732 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.566772 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260928 17:51:42.566917 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.562582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.0.98.2:45443)
0928 17:51:42.562681 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.562686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.562828 (+ 142us) server_negotiation.cc:407] Connection header received
0928 17:51:42.562921 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.562924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.562938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.563043 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:42.563212 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564509 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.565715 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566428 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566499 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566551 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.566766 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.566818 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":49,"threads_started":1}
I20260928 17:51:42.567479 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7d57dd1884672bb2f8ff4d5a87a65 (DEFAULT_TABLE table=test-table [id=6c3e73c813034f6aafa87751a51aa424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:42.567493 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7d57dd1884672bb2f8ff4d5a87a65 (DEFAULT_TABLE table=test-table [id=6c3e73c813034f6aafa87751a51aa424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:42.567796 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7d57dd1884672bb2f8ff4d5a87a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:42.567811 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7d57dd1884672bb2f8ff4d5a87a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:42.568228 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.562754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42091 (local address 127.0.0.1:35744)
0928 17:51:42.562943 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.562954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.562993 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.563569 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.563570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.563571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:42.563613 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.563614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564808 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.564808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.565679 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.565682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566587 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.566654 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566688 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.566732 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.566772 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":29,"thread_start_us":67,"threads_started":1}
I20260928 17:51:42.566941 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.563148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.0.98.3:42091)
0928 17:51:42.563427 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.563431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.563441 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:51:42.563476 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.563478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.563492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.563575 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:42.563686 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.564704 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.565765 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.566434 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.566499 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.566551 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.566815 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.566853 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260928 17:51:42.568924 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7d57dd1884672bb2f8ff4d5a87a65 (DEFAULT_TABLE table=test-table [id=6c3e73c813034f6aafa87751a51aa424]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:42.569239 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7d57dd1884672bb2f8ff4d5a87a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:42.570434 1440 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Bootstrap starting.
I20260928 17:51:42.570983 1441 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: Bootstrap starting.
I20260928 17:51:42.571051 1440 tablet_bootstrap.cc:654] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:42.571376 1440 log.cc:824] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:42.571645 1441 tablet_bootstrap.cc:654] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:42.572137 1440 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: No bootstrap required, opened a new log
I20260928 17:51:42.572206 1440 ts_tablet_manager.cc:1397] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:42.572430 1441 log.cc:824] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:42.572562 1443 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: Bootstrap starting.
I20260928 17:51:42.573069 1443 tablet_bootstrap.cc:654] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:42.573525 1443 log.cc:824] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:42.574165 1441 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: No bootstrap required, opened a new log
I20260928 17:51:42.574245 1441 ts_tablet_manager.cc:1397] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 17:51:42.574258 1443 tablet_bootstrap.cc:492] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: No bootstrap required, opened a new log
I20260928 17:51:42.574333 1443 ts_tablet_manager.cc:1397] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:51:42.574373 1440 raft_consensus.cc:348] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.574535 1440 raft_consensus.cc:374] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:42.574576 1440 raft_consensus.cc:729] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2f447d53a54c3f9c7adde292198e0f, State: Initialized, Role: FOLLOWER
I20260928 17:51:42.574728 1440 consensus_queue.cc:260] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.574977 1440 ts_tablet_manager.cc:1428] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:51:42.575093 1153 heartbeater.cc:503] Master 127.0.98.62:42199 was elected leader, sending a full tablet report...
I20260928 17:51:42.576284 1443 raft_consensus.cc:348] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.576442 1443 raft_consensus.cc:374] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:42.576467 1443 raft_consensus.cc:729] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9480f19fc374f63986dd546fbf9ed1e, State: Initialized, Role: FOLLOWER
I20260928 17:51:42.576582 1443 consensus_queue.cc:260] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.576536 1441 raft_consensus.cc:348] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.576710 1441 raft_consensus.cc:374] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:42.576754 1441 raft_consensus.cc:729] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de349f915ef2402a8bd721be9aeebded, State: Initialized, Role: FOLLOWER
I20260928 17:51:42.576866 1421 heartbeater.cc:503] Master 127.0.98.62:42199 was elected leader, sending a full tablet report...
I20260928 17:51:42.576862 1441 consensus_queue.cc:260] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.576849 1443 ts_tablet_manager.cc:1428] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:51:42.577054 1441 ts_tablet_manager.cc:1428] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 17:51:42.577116 1287 heartbeater.cc:503] Master 127.0.98.62:42199 was elected leader, sending a full tablet report...
W20260928 17:51:42.596048 1154 tablet.cc:2369] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:51:42.635208 1447 raft_consensus.cc:482] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:42.635460 1447 raft_consensus.cc:504] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.635928 1447 leader_election.cc:288] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de349f915ef2402a8bd721be9aeebded (127.0.98.2:45443), 2c2f447d53a54c3f9c7adde292198e0f (127.0.98.1:35941)
I20260928 17:51:42.639235 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.636328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60565 (local address 127.0.98.2:45443)
0928 17:51:42.636378 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.636380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.636495 (+ 115us) server_negotiation.cc:407] Connection header received
0928 17:51:42.636616 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.636618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.636631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.636684 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:42.636865 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.637315 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638465 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.638832 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638888 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.638898 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.639118 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.639160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:51:42.639245 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.636222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35941 (local address 127.0.98.3:54059)
0928 17:51:42.636321 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.636336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.636409 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.636645 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.636646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.636648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:42.636722 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.636724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.637471 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.637472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.638302 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.638304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.638970 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.639030 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.639033 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.639119 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.639163 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":60}
I20260928 17:51:42.639257 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.636444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54059 (local address 127.0.98.1:35941)
0928 17:51:42.636490 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.636493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.636515 (+ 22us) server_negotiation.cc:407] Connection header received
0928 17:51:42.636545 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.636546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.636559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.636617 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:42.636787 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.637351 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638416 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.638853 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638959 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.638969 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.639146 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.639189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:51:42.639268 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.636233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45443 (local address 127.0.98.3:60565)
0928 17:51:42.636431 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:42.636443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:42.636523 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:42.636700 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:42.636701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:42.636702 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:42.636784 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.636786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.637409 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.637410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.638338 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:42.638341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.638916 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.638917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:42.639039 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.639042 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:51:42.639117 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:51:42.639169 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":34,"thread_start_us":119,"threads_started":1}
I20260928 17:51:42.639613 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7d57dd1884672bb2f8ff4d5a87a65" candidate_uuid: "f9480f19fc374f63986dd546fbf9ed1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2f447d53a54c3f9c7adde292198e0f" is_pre_election: true
I20260928 17:51:42.639791 1107 raft_consensus.cc:2454] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9480f19fc374f63986dd546fbf9ed1e in term 0.
I20260928 17:51:42.639881 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7d57dd1884672bb2f8ff4d5a87a65" candidate_uuid: "f9480f19fc374f63986dd546fbf9ed1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de349f915ef2402a8bd721be9aeebded" is_pre_election: true
I20260928 17:51:42.640100 1311 leader_election.cc:302] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2f447d53a54c3f9c7adde292198e0f, f9480f19fc374f63986dd546fbf9ed1e; no voters:
I20260928 17:51:42.640072 1241 raft_consensus.cc:2454] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9480f19fc374f63986dd546fbf9ed1e in term 0.
I20260928 17:51:42.640251 1447 raft_consensus.cc:2781] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:42.640323 1447 raft_consensus.cc:482] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:42.640350 1447 raft_consensus.cc:3037] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:42.641113 1447 raft_consensus.cc:504] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.641264 1447 leader_election.cc:288] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [CANDIDATE]: Term 1 election: Requested vote from peers de349f915ef2402a8bd721be9aeebded (127.0.98.2:45443), 2c2f447d53a54c3f9c7adde292198e0f (127.0.98.1:35941)
I20260928 17:51:42.641417 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7d57dd1884672bb2f8ff4d5a87a65" candidate_uuid: "f9480f19fc374f63986dd546fbf9ed1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2f447d53a54c3f9c7adde292198e0f"
I20260928 17:51:42.641499 1107 raft_consensus.cc:3037] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:42.642664 1107 raft_consensus.cc:2454] T 09b7d57dd1884672bb2f8ff4d5a87a65 P 2c2f447d53a54c3f9c7adde292198e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9480f19fc374f63986dd546fbf9ed1e in term 1.
I20260928 17:51:42.642774 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7d57dd1884672bb2f8ff4d5a87a65" candidate_uuid: "f9480f19fc374f63986dd546fbf9ed1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de349f915ef2402a8bd721be9aeebded"
I20260928 17:51:42.642903 1311 leader_election.cc:302] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2f447d53a54c3f9c7adde292198e0f, f9480f19fc374f63986dd546fbf9ed1e; no voters:
I20260928 17:51:42.642877 1241 raft_consensus.cc:3037] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:42.643000 1447 raft_consensus.cc:2781] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:42.643175 1447 raft_consensus.cc:686] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [term 1 LEADER]: Becoming Leader. State: Replica: f9480f19fc374f63986dd546fbf9ed1e, State: Running, Role: LEADER
I20260928 17:51:42.643287 1447 consensus_queue.cc:237] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } }
I20260928 17:51:42.643918 1241 raft_consensus.cc:2454] T 09b7d57dd1884672bb2f8ff4d5a87a65 P de349f915ef2402a8bd721be9aeebded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9480f19fc374f63986dd546fbf9ed1e in term 1.
I20260928 17:51:42.644171 962 catalog_manager.cc:5495] T 09b7d57dd1884672bb2f8ff4d5a87a65 P f9480f19fc374f63986dd546fbf9ed1e reported cstate change: term changed from 0 to 1, leader changed from <none> to f9480f19fc374f63986dd546fbf9ed1e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f9480f19fc374f63986dd546fbf9ed1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de349f915ef2402a8bd721be9aeebded" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2f447d53a54c3f9c7adde292198e0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9480f19fc374f63986dd546fbf9ed1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42091 } health_report { overall_health: HEALTHY } } }
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:42.664029 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.656465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.0.98.1:35941)
0928 17:51:42.656530 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.656533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.656640 (+ 107us) server_negotiation.cc:407] Connection header received
0928 17:51:42.656744 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.656745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.656755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.656821 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:42.657409 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.657898 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.658639 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.658845 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.658925 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.661363 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:42.661378 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:42.661378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:42.661393 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:42.662824 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.663169 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.663173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.663174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.663234 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.663438 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.663439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.663439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.663586 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:42.663773 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.663810 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.663887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:42.686822 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.679619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.98.1:35941)
0928 17:51:42.679705 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.679707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.679789 (+ 82us) server_negotiation.cc:407] Connection header received
0928 17:51:42.679886 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.679887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.679897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.679963 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:42.680618 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.681113 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.681860 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.682066 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.682152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.684340 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:42.684341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:42.684342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:42.684350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:42.685573 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.685925 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.685926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.685927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.685985 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.686198 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.686199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.686199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.686306 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:42.686405 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.686604 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.686711 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:42.708066 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:42.701801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.0.98.1:35941)
0928 17:51:42.701873 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:42.701875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:42.701888 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:51:42.701960 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:42.701961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:42.701969 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:42.702033 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:42.702529 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.702990 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.703683 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:42.703864 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:42.703916 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:42.705991 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:42.705992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:42.705993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:42.705999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:42.707102 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.707389 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.707390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.707390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.707435 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:42.707664 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:42.707664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:42.707665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:42.707762 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:42.707833 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:42.707946 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 17:51:42.708001 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 17:51:42.708619 1067 tablet_service.cc:2908] Rejecting scan request for tablet 09b7d57dd1884672bb2f8ff4d5a87a65: Uninitialized: clean time has not yet been initialized
I20260928 17:51:42.709467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1024
I20260928 17:51:42.714519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1158
I20260928 17:51:42.719473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1292
I20260928 17:51:42.724290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 930
2026-09-28T17:51:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2598 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[1464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[1464](info): set up 2 sockets
Sep 28 17:51:42 dist-test-slave-zqlt krb5kdc[1464](info): commencing operation
krb5kdc: starting...
W20260928 17:51:43.759769 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:43 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:43.875043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:43.956051 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:43.956338 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:43.956382 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:43.957599 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:43.957679 1477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:51:43.957702 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:43.957744 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:43.957775 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:43.957798 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:43.959383 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:43.959694 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:43.959966 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:43.964980 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:43.965003 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:43.964977 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:43.965159 1477 server_base.cc:1034] running on GCE node
I20260928 17:51:43.965451 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:43.965770 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:43.966953 1477 hybrid_clock.cc:648] HybridClock initialized: now 1790617903966937 us; error 37 us; skew 500 ppm
Sep 28 17:51:43 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:43.969712 1477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:43.970163 1477 webserver.cc:460] Webserver started at http://127.0.98.62:34303/ using document root <none> and password file <none>
I20260928 17:51:43.970414 1477 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:43.970463 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:43.970594 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:43.971688 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "a62025f68ca44943a944c24a8c09f206"
format_stamp: "Formatted at 2026-09-28 17:51:43 on dist-test-slave-zqlt"
server_key: "ea127acdeb61fd3ca0dcaf3279ceaf4c"
server_key_iv: "3ab085d9c852b68ebe34421568d9505c"
server_key_version: "encryptionkey@0"
I20260928 17:51:43.972056 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "a62025f68ca44943a944c24a8c09f206"
format_stamp: "Formatted at 2026-09-28 17:51:43 on dist-test-slave-zqlt"
server_key: "ea127acdeb61fd3ca0dcaf3279ceaf4c"
server_key_iv: "3ab085d9c852b68ebe34421568d9505c"
server_key_version: "encryptionkey@0"
I20260928 17:51:43.973465 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:43.974200 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:43.974385 1477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:43.974488 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "a62025f68ca44943a944c24a8c09f206"
format_stamp: "Formatted at 2026-09-28 17:51:43 on dist-test-slave-zqlt"
server_key: "ea127acdeb61fd3ca0dcaf3279ceaf4c"
server_key_iv: "3ab085d9c852b68ebe34421568d9505c"
server_key_version: "encryptionkey@0"
I20260928 17:51:43.974562 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:43.986644 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:43.989020 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:43.989178 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:43.993603 1477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44799
I20260928 17:51:43.993671 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44799 every 8 connection(s)
I20260928 17:51:43.994019 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:43.994710 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:43.997009 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Bootstrap starting.
I20260928 17:51:43.997591 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:43.997888 1555 log.cc:824] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:43.998521 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: No bootstrap required, opened a new log
I20260928 17:51:43.999982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1477
I20260928 17:51:44.000084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:44.000262 392 external_mini_cluster.cc:1428] Setting key c03850e7c14bd7168af6851853e48566
I20260928 17:51:44.000856 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } }
I20260928 17:51:44.001047 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.001084 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62025f68ca44943a944c24a8c09f206, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.001202 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } }
I20260928 17:51:44.001278 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:44.001323 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:44.001374 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.001879 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } }
I20260928 17:51:44.001993 1555 leader_election.cc:302] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a62025f68ca44943a944c24a8c09f206; no voters:
I20260928 17:51:44.002159 1555 leader_election.cc:288] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:44.002230 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:44.002380 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:44.002389 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [term 1 LEADER]: Becoming Leader. State: Replica: a62025f68ca44943a944c24a8c09f206, State: Running, Role: LEADER
I20260928 17:51:44.002525 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } }
I20260928 17:51:44.002887 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a62025f68ca44943a944c24a8c09f206. Latest consensus state: current_term: 1 leader_uuid: "a62025f68ca44943a944c24a8c09f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } } }
I20260928 17:51:44.002964 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:44.003113 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a62025f68ca44943a944c24a8c09f206" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62025f68ca44943a944c24a8c09f206" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44799 } } }
I20260928 17:51:44.003176 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:44.003144 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:44.003806 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:44.006325 1569 catalog_manager.cc:1347] Generated new cluster ID: c82a11a5bd9e4a6f9a7b5d291d871683
I20260928 17:51:44.006376 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.012290 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.000798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.98.62:44799)
0928 17:51:44.000988 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.000997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.001029 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:51:44.001149 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.001156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.001210 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.001365 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.001993 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.003527 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.004651 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.005253 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.005358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.008039 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.008044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.008044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.008056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.010607 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.011006 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.011009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.011010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.011129 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.011339 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.011340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.011340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.011654 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.011827 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.011910 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.012043 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 17:51:44.022576 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:44.023125 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:44.028069 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Generated new TSK 0
I20260928 17:51:44.028295 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:44.062805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44799
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:44.151255 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:44.151495 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:44.151539 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:44.152835 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:44.152904 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:44.152940 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:44.152977 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:44.154547 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:44.154821 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:44.155056 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:44.160024 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:44.160053 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:44.160143 1581 server_base.cc:1034] running on GCE node
W20260928 17:51:44.160024 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:44.160552 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:44.160866 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:44.162053 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790617904162044 us; error 39 us; skew 500 ppm
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:44.164861 1581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:44.165310 1581 webserver.cc:460] Webserver started at http://127.0.98.1:42765/ using document root <none> and password file <none>
I20260928 17:51:44.165558 1581 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:44.165630 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:44.165784 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:44.166878 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3c5db8845544cad9954688b9c4a3df9"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "abb01607dd272a09466148d7f2d13319"
server_key_iv: "556b6314bd40384260106624c37c4094"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.167259 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3c5db8845544cad9954688b9c4a3df9"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "abb01607dd272a09466148d7f2d13319"
server_key_iv: "556b6314bd40384260106624c37c4094"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.168828 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:51:44.169637 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.169832 1581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.169931 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "b3c5db8845544cad9954688b9c4a3df9"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "abb01607dd272a09466148d7f2d13319"
server_key_iv: "556b6314bd40384260106624c37c4094"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.170017 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:44.194684 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:44.196763 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:44.196938 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:44.197464 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:44.197533 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.197582 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:44.197608 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.204484 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41371
I20260928 17:51:44.204561 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41371 every 8 connection(s)
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.204909 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:51:44.207755 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.198007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58571 (local address 127.0.98.62:44799)
0928 17:51:44.198081 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.198083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.198270 (+ 187us) server_negotiation.cc:407] Connection header received
0928 17:51:44.198805 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.198806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.198823 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.198887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.199756 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.200642 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.202233 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.202480 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.202569 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.205253 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.205255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.205256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.205263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.206646 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.207007 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.207008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.207008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.207058 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.207280 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.207280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.207280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.207376 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.207425 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.207631 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.207693 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:51:44.207928 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.197986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.1:58571)
0928 17:51:44.198146 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.198185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.198691 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.198959 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.198963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.199332 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.199601 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.199608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.200746 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.200748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.202129 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.202132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.202578 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.202579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.202723 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.203252 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.203263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.205140 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.206725 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.206731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.206733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.206939 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.207114 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.207115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.207115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.207225 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.207429 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.207433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.207540 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.207618 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.207735 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260928 17:51:44.208516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1581
I20260928 17:51:44.208521 1712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44799
I20260928 17:51:44.208601 1712 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:44.208627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:44.208792 1712 heartbeater.cc:511] Master 127.0.98.62:44799 requested a full tablet report, sending...
I20260928 17:51:44.208799 392 external_mini_cluster.cc:1428] Setting key 819a3c2df70d00236c4b62fdd8fb1933
I20260928 17:51:44.209338 1508 ts_manager.cc:194] Registered new tserver with Master: b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371)
I20260928 17:51:44.210153 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58571
I20260928 17:51:44.215682 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.209552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43847 (local address 127.0.98.62:44799)
0928 17:51:44.209596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.209598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.209776 (+ 178us) server_negotiation.cc:407] Connection header received
0928 17:51:44.209818 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.209819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.209829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.209877 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.210239 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.210897 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.211600 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.211923 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.211992 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.213060 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.213061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.213062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.213067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.214469 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.214898 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.214899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.214899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.214959 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.215173 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.215174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.215174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.215301 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.215359 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.215516 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.215616 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:51:44.215739 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.209554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.1:43847)
0928 17:51:44.209697 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.209708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.209767 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.209881 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.209882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.210097 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.210161 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.210163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.211012 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.211013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.211518 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.211520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.212012 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.212013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.212121 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.212502 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.212507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.212973 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.214556 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.214557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.214557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.214799 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.215037 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.215038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.215038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.215106 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.215370 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.215370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.215420 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.215498 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.215623 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:44.234323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44799
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:44.318019 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:44.318245 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:44.318274 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:44.319535 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:44.319585 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:44.319607 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:44.319625 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:44.321223 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:44.321422 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:44.321611 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:44.326620 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:44.326634 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:44.326678 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:44.326866 1721 server_base.cc:1034] running on GCE node
I20260928 17:51:44.327073 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:44.327369 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:44.328553 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790617904328543 us; error 43 us; skew 500 ppm
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:44.331453 1721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:44.331873 1721 webserver.cc:460] Webserver started at http://127.0.98.2:42619/ using document root <none> and password file <none>
I20260928 17:51:44.332077 1721 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:44.332190 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:44.332314 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:44.333367 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea262c58010e409fa4ec3439bd838288"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "870fd04521f83f7a00fbac808960b72d"
server_key_iv: "51ed0d3d9cd598eaf382b7542db9483a"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.333675 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea262c58010e409fa4ec3439bd838288"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "870fd04521f83f7a00fbac808960b72d"
server_key_iv: "51ed0d3d9cd598eaf382b7542db9483a"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.334976 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:44.335688 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.335871 1721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:44.335958 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "ea262c58010e409fa4ec3439bd838288"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "870fd04521f83f7a00fbac808960b72d"
server_key_iv: "51ed0d3d9cd598eaf382b7542db9483a"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.336015 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:44.351814 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:44.353952 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:44.354071 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:44.354557 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:44.354596 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.354619 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:44.354632 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.361891 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35867
I20260928 17:51:44.361959 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35867 every 8 connection(s)
I20260928 17:51:44.362298 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.365985 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.355162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45639 (local address 127.0.98.62:44799)
0928 17:51:44.355233 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.355236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.355449 (+ 213us) server_negotiation.cc:407] Connection header received
0928 17:51:44.356011 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.356012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.356027 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.356147 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.357124 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.357997 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.359799 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.360044 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.360165 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.362911 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.362913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.362914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.362922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.364514 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.364960 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.364962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.364962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.365027 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.365292 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.365293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.365293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.365418 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.365486 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.365807 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.365893 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:51:44.366187 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.355137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.2:45639)
0928 17:51:44.355310 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.355358 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.355864 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.356273 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.356278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.356709 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.357008 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.357016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.358125 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.358126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.359661 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.359664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.360180 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.360182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.360352 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.360991 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.361003 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.362809 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.364626 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.364633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.364635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.364865 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.365105 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.365106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.365106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.365217 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.365489 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.365495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.365658 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.365771 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.365938 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260928 17:51:44.366704 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44799
I20260928 17:51:44.366812 1852 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:44.367007 1852 heartbeater.cc:511] Master 127.0.98.62:44799 requested a full tablet report, sending...
I20260928 17:51:44.367481 1509 ts_manager.cc:194] Registered new tserver with Master: ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867)
I20260928 17:51:44.368019 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45639
I20260928 17:51:44.369273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1721
I20260928 17:51:44.369344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:44.369524 392 external_mini_cluster.cc:1428] Setting key ad25fa6f0bd215502ad186aaa34a9d07
I20260928 17:51:44.373849 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.367844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52411 (local address 127.0.98.62:44799)
0928 17:51:44.367887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.367889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.367992 (+ 103us) server_negotiation.cc:407] Connection header received
0928 17:51:44.368063 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.368063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.368071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.368189 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.368561 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.369029 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.369825 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.370013 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.370124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.371063 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.371064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.371065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.371072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.372673 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.373025 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.373026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.373027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.373083 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.373291 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.373292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.373292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.373441 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.373516 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.373673 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.373753 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:51:44.373865 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.367758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.2:52411)
0928 17:51:44.367935 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.367946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.368006 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.368193 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.368195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.368423 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.368491 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.368493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.369117 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.369118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.369731 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.369734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.370126 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.370127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.370192 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.370684 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.370688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.370971 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.372760 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.372761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.372762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.372939 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.373158 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.373160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.373160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.373218 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.373517 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.373519 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.373565 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.373650 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.373760 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:44.394601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44799
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:44.480773 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:44.480969 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:44.480993 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:44.482074 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:44.482106 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:44.482125 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:44.482143 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:44.483543 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:44.483738 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:44.483906 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:44.488981 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:44.488955 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:44.488960 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:44.489212 1861 server_base.cc:1034] running on GCE node
I20260928 17:51:44.489421 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:44.489631 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:44.490798 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790617904490792 us; error 40 us; skew 500 ppm
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:44.493358 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:44.493796 1861 webserver.cc:460] Webserver started at http://127.0.98.3:42465/ using document root <none> and password file <none>
I20260928 17:51:44.494040 1861 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:44.494107 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:44.494261 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:44.495307 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec1c969cac484ee7970179b2bf5c9d47"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "36f724491449959033d7d08a9ec9634f"
server_key_iv: "d9f95f65b7a4370c3448281e7f9422da"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.495680 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec1c969cac484ee7970179b2bf5c9d47"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "36f724491449959033d7d08a9ec9634f"
server_key_iv: "d9f95f65b7a4370c3448281e7f9422da"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.497092 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:44.497768 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.497922 1861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.498027 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "ec1c969cac484ee7970179b2bf5c9d47"
format_stamp: "Formatted at 2026-09-28 17:51:44 on dist-test-slave-zqlt"
server_key: "36f724491449959033d7d08a9ec9634f"
server_key_iv: "d9f95f65b7a4370c3448281e7f9422da"
server_key_version: "encryptionkey@0"
I20260928 17:51:44.498101 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:44.506227 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:44.508590 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:44.508772 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:44.509239 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:44.509302 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.509346 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:44.509372 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.516548 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44003
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.516633 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44003 every 8 connection(s)
I20260928 17:51:44.516968 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:44.519241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 1861
I20260928 17:51:44.519327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:44.519505 392 external_mini_cluster.cc:1428] Setting key 1cdd0e633e63bfba19fdfaa0b4e34965
I20260928 17:51:44.519934 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.509737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33821 (local address 127.0.98.62:44799)
0928 17:51:44.509788 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.509790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.510041 (+ 251us) server_negotiation.cc:407] Connection header received
0928 17:51:44.510792 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.510793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.510804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.510856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.511772 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.512632 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.514339 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.514551 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.514655 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.517154 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.517156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.517156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.517166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.518521 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.518925 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.518925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.518926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.518979 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.519199 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.519200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.519201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.519355 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.519433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.519734 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.519827 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:51:44.520026 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.509708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.3:33821)
0928 17:51:44.509889 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.509943 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.510662 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.510942 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.510947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.511360 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.511668 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.511676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.512752 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.512754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.514242 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.514246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.514664 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.514666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.514796 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.515309 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.515323 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.517056 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.518624 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.518629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.518631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.518852 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.519036 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.519037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.519037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.519135 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.519420 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.519425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.519585 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.519699 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.519837 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260928 17:51:44.520543 1992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44799
I20260928 17:51:44.520633 1992 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:44.520824 1992 heartbeater.cc:511] Master 127.0.98.62:44799 requested a full tablet report, sending...
I20260928 17:51:44.521216 1509 ts_manager.cc:194] Registered new tserver with Master: ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003)
I20260928 17:51:44.521629 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33821
I20260928 17:51:44.527133 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.521694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37865 (local address 127.0.98.62:44799)
0928 17:51:44.521737 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.521739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.521883 (+ 144us) server_negotiation.cc:407] Connection header received
0928 17:51:44.521945 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.521946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.521956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.522011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.522341 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.522827 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.523544 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.523751 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.523838 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.524853 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.524854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.524855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.524862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.525966 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.526335 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.526336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.526336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.526393 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.526587 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.526588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.526589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.526717 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.526793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.526956 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.527037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:51:44.527146 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.521618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.98.3:37865)
0928 17:51:44.521823 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.521833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.521897 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.522017 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.522018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.522217 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.522267 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.522269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.522946 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.522947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.523457 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.523459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.523839 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.523840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.523905 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.524438 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.524442 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.524726 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.526065 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.526066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.526066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.526249 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.526461 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.526461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.526462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.526524 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.526796 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.526797 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.526841 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.526930 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.527039 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260928 17:51:44.531394 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.553800 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.546989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.0.98.62:44799)
0928 17:51:44.547060 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.547062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.547077 (+ 15us) server_negotiation.cc:407] Connection header received
0928 17:51:44.547100 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.547101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.547110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.547176 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.547756 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.548337 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.549088 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.549289 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.549378 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.551582 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.551584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.551584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.551592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.552861 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.553175 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.553176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.553176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.553240 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.553399 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.553400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.553400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.553497 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.553571 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.553670 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.553730 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:51:44.555433 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:44.557188 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:44.563639 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.0.1:55710)
0928 17:51:44.559610 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.559652 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.559752 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.560104 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.560107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.560110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.560199 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.560201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561231 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.561232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.562383 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.562385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.563325 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.563326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.563407 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.563422 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.563497 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.563553 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 17:51:44.563649 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.98.2:35867)
0928 17:51:44.559718 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.559724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.559737 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:51:44.559895 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.559900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.559919 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.559994 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.560287 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561133 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.562472 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.563219 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.563321 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.563407 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.563511 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.563551 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260928 17:51:44.564028 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41371 (local address 127.0.0.1:51286)
0928 17:51:44.560152 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.560161 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.560214 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.560487 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.560488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.560490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.560525 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.560527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561762 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.561763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.562758 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.562760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.563778 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.563780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.563855 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.563860 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.563916 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.563956 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
I20260928 17:51:44.564141 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.0.98.1:41371)
0928 17:51:44.560068 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.560072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.560282 (+ 210us) server_negotiation.cc:407] Connection header received
0928 17:51:44.560330 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.560333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.560350 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.560419 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.560610 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561682 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.562845 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.563593 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.563665 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.563737 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.563962 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.564015 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20260928 17:51:44.564486 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a (DEFAULT_TABLE table=test-table [id=1571f9cbe8a54a5cbdd698b204f76896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.564718 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44003 (local address 127.0.0.1:45364)
0928 17:51:44.559990 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.560001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.560054 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.560330 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.560331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.560334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.560391 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.560393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561784 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.561785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.562941 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.562944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.564476 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.564477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.564553 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.564557 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.564612 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.564652 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":86,"threads_started":1}
I20260928 17:51:44.564888 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f790883d6894e51bcef7eb9e5e0416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.565048 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.559810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.0.98.3:44003)
0928 17:51:44.559997 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.560002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.560048 (+ 46us) server_negotiation.cc:407] Connection header received
0928 17:51:44.560152 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.560154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.560170 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.560449 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.560510 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.561672 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.563029 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.564326 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.564417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.564493 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.564903 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.564952 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260928 17:51:44.565699 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a (DEFAULT_TABLE table=test-table [id=1571f9cbe8a54a5cbdd698b204f76896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.566061 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f790883d6894e51bcef7eb9e5e0416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.568647 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a (DEFAULT_TABLE table=test-table [id=1571f9cbe8a54a5cbdd698b204f76896]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.568982 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f790883d6894e51bcef7eb9e5e0416a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.569864 2016 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Bootstrap starting.
I20260928 17:51:44.570592 2016 tablet_bootstrap.cc:654] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.570914 2016 log.cc:824] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:44.571739 2016 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: No bootstrap required, opened a new log
I20260928 17:51:44.571808 2016 ts_tablet_manager.cc:1397] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:51:44.572369 2015 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Bootstrap starting.
I20260928 17:51:44.572773 2018 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Bootstrap starting.
I20260928 17:51:44.573057 2015 tablet_bootstrap.cc:654] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.573411 2015 log.cc:824] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:44.573457 2018 tablet_bootstrap.cc:654] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.573788 2018 log.cc:824] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:44.574016 2016 raft_consensus.cc:348] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.574218 2016 raft_consensus.cc:374] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.574270 2016 raft_consensus.cc:729] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1c969cac484ee7970179b2bf5c9d47, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.574303 2015 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: No bootstrap required, opened a new log
I20260928 17:51:44.574417 2015 ts_tablet_manager.cc:1397] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:44.574402 2016 consensus_queue.cc:260] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.574499 2018 tablet_bootstrap.cc:492] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: No bootstrap required, opened a new log
I20260928 17:51:44.574568 2018 ts_tablet_manager.cc:1397] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:51:44.574687 2016 ts_tablet_manager.cc:1428] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:51:44.574734 1992 heartbeater.cc:503] Master 127.0.98.62:44799 was elected leader, sending a full tablet report...
I20260928 17:51:44.576553 2018 raft_consensus.cc:348] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.576713 2018 raft_consensus.cc:374] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.576738 2018 raft_consensus.cc:729] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c5db8845544cad9954688b9c4a3df9, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.576861 2018 consensus_queue.cc:260] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.577122 2018 ts_tablet_manager.cc:1428] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:51:44.577147 1712 heartbeater.cc:503] Master 127.0.98.62:44799 was elected leader, sending a full tablet report...
I20260928 17:51:44.577366 2015 raft_consensus.cc:348] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.577527 2015 raft_consensus.cc:374] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.577559 2015 raft_consensus.cc:729] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea262c58010e409fa4ec3439bd838288, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.577690 2015 consensus_queue.cc:260] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.578086 2015 ts_tablet_manager.cc:1428] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:51:44.578086 1852 heartbeater.cc:503] Master 127.0.98.62:44799 was elected leader, sending a full tablet report...
W20260928 17:51:44.612922 1853 tablet.cc:2369] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:51:44.633852 2022 raft_consensus.cc:482] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:44.633973 2022 raft_consensus.cc:504] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.634276 2022 leader_election.cc:288] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867), ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003)
I20260928 17:51:44.637176 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.634590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40091 (local address 127.0.98.3:44003)
0928 17:51:44.634633 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.634635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.634794 (+ 159us) server_negotiation.cc:407] Connection header received
0928 17:51:44.634821 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.634822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.634836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.634890 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.635063 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.635526 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636478 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.636830 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636882 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.636890 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.637065 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.637104 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:51:44.637202 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.634575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34739 (local address 127.0.98.2:35867)
0928 17:51:44.634617 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.634619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.634645 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:51:44.634674 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.634674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.634686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.634758 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.634940 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.635459 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636475 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.636845 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636938 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.636948 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.637103 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.637138 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:51:44.637219 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.634472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44003 (local address 127.0.98.1:40091)
0928 17:51:44.634697 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.634706 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.634754 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.634922 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.634923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.634924 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.634993 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.634994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.635611 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.635612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.636380 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.636382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.636909 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.636987 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.636991 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.637041 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.637093 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":57,"thread_start_us":108,"threads_started":1}
I20260928 17:51:44.637255 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.634472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.98.1:34739)
0928 17:51:44.634536 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.634550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.634606 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.634783 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.634784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.634786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.634864 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.634866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.635542 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.635543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.636378 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.636380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.636958 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.636959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.637012 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.637015 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.637088 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.637129 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260928 17:51:44.637605 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f790883d6894e51bcef7eb9e5e0416a" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288" is_pre_election: true
I20260928 17:51:44.637656 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f790883d6894e51bcef7eb9e5e0416a" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1c969cac484ee7970179b2bf5c9d47" is_pre_election: true
I20260928 17:51:44.637804 1806 raft_consensus.cc:2454] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c5db8845544cad9954688b9c4a3df9 in term 0.
I20260928 17:51:44.637813 1946 raft_consensus.cc:2454] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c5db8845544cad9954688b9c4a3df9 in term 0.
I20260928 17:51:44.638041 1600 leader_election.cc:302] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5db8845544cad9954688b9c4a3df9, ea262c58010e409fa4ec3439bd838288; no voters:
I20260928 17:51:44.638180 2022 raft_consensus.cc:2781] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:44.638232 2022 raft_consensus.cc:482] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:44.638259 2022 raft_consensus.cc:3037] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.639299 2022 raft_consensus.cc:504] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.639438 2022 leader_election.cc:288] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 election: Requested vote from peers ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867), ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003)
I20260928 17:51:44.639572 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f790883d6894e51bcef7eb9e5e0416a" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288"
I20260928 17:51:44.639602 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f790883d6894e51bcef7eb9e5e0416a" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1c969cac484ee7970179b2bf5c9d47"
I20260928 17:51:44.639657 1946 raft_consensus.cc:3037] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.639669 1806 raft_consensus.cc:3037] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.640362 1946 raft_consensus.cc:2454] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c5db8845544cad9954688b9c4a3df9 in term 1.
I20260928 17:51:44.640475 1806 raft_consensus.cc:2454] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c5db8845544cad9954688b9c4a3df9 in term 1.
I20260928 17:51:44.640537 1598 leader_election.cc:302] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5db8845544cad9954688b9c4a3df9, ec1c969cac484ee7970179b2bf5c9d47; no voters:
I20260928 17:51:44.640625 2022 raft_consensus.cc:2781] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:44.640815 2022 raft_consensus.cc:686] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 1 LEADER]: Becoming Leader. State: Replica: b3c5db8845544cad9954688b9c4a3df9, State: Running, Role: LEADER
I20260928 17:51:44.640923 2022 consensus_queue.cc:237] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } }
I20260928 17:51:44.641674 1509 catalog_manager.cc:5495] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b3c5db8845544cad9954688b9c4a3df9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } health_report { overall_health: HEALTHY } } }
I20260928 17:51:44.656845 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.654251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.0.98.1:41371)
0928 17:51:44.654312 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.654315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.654323 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:51:44.654440 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.654441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.654468 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.654543 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:44.654687 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.655148 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.655957 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.656179 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.656253 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.656469 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.656578 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.656672 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.656727 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.656753 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:51:44.658545 1806 raft_consensus.cc:1261] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Refusing update from remote peer b3c5db8845544cad9954688b9c4a3df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.658573 1946 raft_consensus.cc:1261] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Refusing update from remote peer b3c5db8845544cad9954688b9c4a3df9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.658869 2022 consensus_queue.cc:1035] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:44.659035 2025 consensus_queue.cc:1035] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:44.661082 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334370937478598656 to 7334370937409761280. Current Snapshot: MvccSnapshot[applied={T|T < 7334370937478598656}]
I20260928 17:51:44.661511 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334370937478598656 to 7334370937409761280. Current Snapshot: MvccSnapshot[applied={T|T < 7334370937478598656}]
I20260928 17:51:44.662038 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334370937478598656 to 7334370937409761280. Current Snapshot: MvccSnapshot[applied={T|T < 7334370937478598656}]
I20260928 17:51:44.664602 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:44.664709 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:44.665472 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Sending DeleteTablet for 3 replicas of tablet 6f790883d6894e51bcef7eb9e5e0416a
I20260928 17:51:44.665848 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45364
I20260928 17:51:44.665921 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55710
I20260928 17:51:44.666639 2039 tablet_replica.cc:330] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: stopping tablet replica
I20260928 17:51:44.666641 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f790883d6894e51bcef7eb9e5e0416a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51286
I20260928 17:51:44.666819 2039 raft_consensus.cc:2229] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:44.666872 2040 tablet_replica.cc:330] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: stopping tablet replica
I20260928 17:51:44.666970 2040 raft_consensus.cc:2229] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:44.667068 2040 raft_consensus.cc:2258] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:44.667083 2039 raft_consensus.cc:2258] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:44.667126 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:44.667271 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:44.667536 2038 tablet_replica.cc:330] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: stopping tablet replica
I20260928 17:51:44.667623 2038 raft_consensus.cc:2229] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:44.667634 2039 ts_tablet_manager.cc:1905] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:44.667985 2038 raft_consensus.cc:2258] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:44.668768 2038 ts_tablet_manager.cc:1905] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:44.669015 1646 tablet_service.cc:1467] Processing CreateTablet for tablet e48f785bf5a74a2597df2c9a8ea996af (DEFAULT_TABLE table=test-table [id=90ece48c39e14809ad9765b3e2808f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.669128 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f785bf5a74a2597df2c9a8ea996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.669400 2039 ts_tablet_manager.cc:1918] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:51:44.669466 2039 log.cc:1196] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/6f790883d6894e51bcef7eb9e5e0416a
I20260928 17:51:44.669723 2039 ts_tablet_manager.cc:1939] T 6f790883d6894e51bcef7eb9e5e0416a P ea262c58010e409fa4ec3439bd838288: Deleting consensus metadata
I20260928 17:51:44.670516 2018 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Bootstrap starting.
I20260928 17:51:44.670547 1926 tablet_service.cc:1467] Processing CreateTablet for tablet e48f785bf5a74a2597df2c9a8ea996af (DEFAULT_TABLE table=test-table [id=90ece48c39e14809ad9765b3e2808f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.670609 2038 ts_tablet_manager.cc:1918] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:51:44.670643 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f785bf5a74a2597df2c9a8ea996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.670677 2038 log.cc:1196] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/6f790883d6894e51bcef7eb9e5e0416a
I20260928 17:51:44.670938 2018 tablet_bootstrap.cc:654] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.670951 2038 ts_tablet_manager.cc:1939] T 6f790883d6894e51bcef7eb9e5e0416a P ec1c969cac484ee7970179b2bf5c9d47: Deleting consensus metadata
I20260928 17:51:44.671341 1494 catalog_manager.cc:4841] TS ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003): tablet 6f790883d6894e51bcef7eb9e5e0416a (table test-table [id=1571f9cbe8a54a5cbdd698b204f76896]) successfully deleted
I20260928 17:51:44.671561 2018 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: No bootstrap required, opened a new log
I20260928 17:51:44.671591 2018 ts_tablet_manager.cc:1397] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.671743 2018 raft_consensus.cc:348] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.671815 2018 raft_consensus.cc:374] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.671839 2018 raft_consensus.cc:729] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c5db8845544cad9954688b9c4a3df9, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.671883 2018 consensus_queue.cc:260] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.671936 2025 raft_consensus.cc:482] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:44.671969 2025 raft_consensus.cc:504] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.672051 2016 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Bootstrap starting.
I20260928 17:51:44.672070 1786 tablet_service.cc:1467] Processing CreateTablet for tablet e48f785bf5a74a2597df2c9a8ea996af (DEFAULT_TABLE table=test-table [id=90ece48c39e14809ad9765b3e2808f92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:44.672127 2025 leader_election.cc:288] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867), ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003)
I20260928 17:51:44.672205 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f785bf5a74a2597df2c9a8ea996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.672415 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1c969cac484ee7970179b2bf5c9d47" is_pre_election: true
I20260928 17:51:44.672555 2016 tablet_bootstrap.cc:654] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Neither blocks nor log segments found. Creating new log.
W20260928 17:51:44.672830 1598 leader_election.cc:341] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003): Illegal state: must be running to vote when last-logged opid is not known
I20260928 17:51:44.673106 2016 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: No bootstrap required, opened a new log
I20260928 17:51:44.673146 2016 ts_tablet_manager.cc:1397] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.671970 2018 ts_tablet_manager.cc:1428] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:44.673300 2016 raft_consensus.cc:348] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.673373 2016 raft_consensus.cc:374] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.673372 2015 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Bootstrap starting.
I20260928 17:51:44.673396 2016 raft_consensus.cc:729] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1c969cac484ee7970179b2bf5c9d47, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.673439 2016 consensus_queue.cc:260] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.673519 2016 ts_tablet_manager.cc:1428] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.673776 2015 tablet_bootstrap.cc:654] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.674216 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "b3c5db8845544cad9954688b9c4a3df9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288" is_pre_election: true
I20260928 17:51:44.674386 1496 catalog_manager.cc:4841] TS ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867): tablet 6f790883d6894e51bcef7eb9e5e0416a (table test-table [id=1571f9cbe8a54a5cbdd698b204f76896]) successfully deleted
I20260928 17:51:44.674422 2015 tablet_bootstrap.cc:492] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: No bootstrap required, opened a new log
I20260928 17:51:44.674465 2015 ts_tablet_manager.cc:1397] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.674636 2015 raft_consensus.cc:348] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.674713 2015 raft_consensus.cc:374] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.674734 2015 raft_consensus.cc:729] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea262c58010e409fa4ec3439bd838288, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.674778 2015 consensus_queue.cc:260] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.674862 2015 ts_tablet_manager.cc:1428] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260928 17:51:44.675071 1600 leader_election.cc:341] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867): Illegal state: must be running to vote when last-logged opid is not known
I20260928 17:51:44.675112 1600 leader_election.cc:302] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3c5db8845544cad9954688b9c4a3df9; no voters: ea262c58010e409fa4ec3439bd838288, ec1c969cac484ee7970179b2bf5c9d47
I20260928 17:51:44.675275 2025 raft_consensus.cc:2726] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 17:51:44.676186 2040 ts_tablet_manager.cc:1905] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:44.677817 2040 ts_tablet_manager.cc:1918] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:51:44.677891 2040 log.cc:1196] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/6f790883d6894e51bcef7eb9e5e0416a
I20260928 17:51:44.678143 2040 ts_tablet_manager.cc:1939] T 6f790883d6894e51bcef7eb9e5e0416a P b3c5db8845544cad9954688b9c4a3df9: Deleting consensus metadata
I20260928 17:51:44.678534 1494 catalog_manager.cc:4841] TS b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371): tablet 6f790883d6894e51bcef7eb9e5e0416a (table test-table [id=1571f9cbe8a54a5cbdd698b204f76896]) successfully deleted
W20260928 17:51:44.705688 1713 tablet.cc:2369] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:51:44.767549 1993 tablet.cc:2369] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:51:44.870297 2021 raft_consensus.cc:482] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:44.870393 2021 raft_consensus.cc:504] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.870710 2021 leader_election.cc:288] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371), ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867)
I20260928 17:51:44.874529 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.870963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56735 (local address 127.0.98.2:35867)
0928 17:51:44.871011 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.871014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.871164 (+ 150us) server_negotiation.cc:407] Connection header received
0928 17:51:44.871197 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.871198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.871207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.871261 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.871419 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.872157 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.873596 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.874172 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.874247 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.874258 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.874421 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.874459 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:51:44.874529 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.870958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39547 (local address 127.0.98.1:41371)
0928 17:51:44.871011 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.871013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.871026 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:51:44.871077 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.871078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.871088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.871165 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:44.871332 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.872078 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.873596 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.874173 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.874248 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.874258 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.874421 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.874457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:51:44.874555 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.870906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41371 (local address 127.0.98.3:39547)
0928 17:51:44.870946 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.870959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.871011 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.871169 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.871170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.871172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.871252 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.871254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.872182 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.872182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.873502 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.873505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.874257 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.874258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.874338 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.874342 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.874404 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.874452 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":63}
I20260928 17:51:44.874567 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.870908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.98.3:56735)
0928 17:51:44.871089 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.871098 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.871153 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.871280 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.871280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.871282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:44.871352 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.871353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.872212 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.872213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.873502 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.873505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.874277 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.874278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.874338 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.874344 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.874404 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.874452 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":88,"thread_start_us":63,"threads_started":1}
I20260928 17:51:44.874868 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288" is_pre_election: true
I20260928 17:51:44.874969 1806 raft_consensus.cc:2454] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 0.
I20260928 17:51:44.874995 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5db8845544cad9954688b9c4a3df9" is_pre_election: true
I20260928 17:51:44.875137 1666 raft_consensus.cc:2454] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 0.
I20260928 17:51:44.875344 1880 leader_election.cc:302] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea262c58010e409fa4ec3439bd838288, ec1c969cac484ee7970179b2bf5c9d47; no voters:
I20260928 17:51:44.875483 2021 raft_consensus.cc:2781] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:44.875524 2021 raft_consensus.cc:482] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:44.875545 2021 raft_consensus.cc:3037] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.876560 2021 raft_consensus.cc:504] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.876729 2021 leader_election.cc:288] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 election: Requested vote from peers b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371), ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867)
I20260928 17:51:44.876855 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288"
I20260928 17:51:44.876864 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f785bf5a74a2597df2c9a8ea996af" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5db8845544cad9954688b9c4a3df9"
I20260928 17:51:44.876935 1806 raft_consensus.cc:3037] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.876964 1666 raft_consensus.cc:3037] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.877626 1806 raft_consensus.cc:2454] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 1.
I20260928 17:51:44.877796 1880 leader_election.cc:302] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea262c58010e409fa4ec3439bd838288, ec1c969cac484ee7970179b2bf5c9d47; no voters:
I20260928 17:51:44.877925 1666 raft_consensus.cc:2454] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 1.
I20260928 17:51:44.878132 2021 raft_consensus.cc:2781] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:44.878278 2021 raft_consensus.cc:686] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 1 LEADER]: Becoming Leader. State: Replica: ec1c969cac484ee7970179b2bf5c9d47, State: Running, Role: LEADER
I20260928 17:51:44.878360 2021 consensus_queue.cc:237] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.879078 1509 catalog_manager.cc:5495] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } health_report { overall_health: HEALTHY } } }
Sep 28 17:51:44 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:44.924343 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.916908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.0.98.62:44799)
0928 17:51:44.916995 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.916998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.917105 (+ 107us) server_negotiation.cc:407] Connection header received
0928 17:51:44.917204 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.917205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.917216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.917305 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:44.917811 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.918412 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.919129 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.919307 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.919392 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.921665 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:44.921666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:44.921666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:44.921674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:44.923081 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.923469 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.923470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.923471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.923527 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:44.923720 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:44.923721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:44.923722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:44.923836 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:44.923916 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.924070 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.924226 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":53}
I20260928 17:51:44.924432 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.916812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44799 (local address 127.0.0.1:38824)
0928 17:51:44.917016 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.917034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.917127 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.917313 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.917314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.917649 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:44.917698 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.917700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.918498 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.918500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.919032 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.919033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.919392 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.919392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.919439 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.920029 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:44.920034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:44.921570 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:44.923157 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.923164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.923166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.923357 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.923594 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:44.923595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:44.923595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:44.923655 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:44.923905 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:44.923913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:44.923970 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.924048 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.924288 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":102,"thread_start_us":118,"threads_started":1}
I20260928 17:51:44.926520 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:51:44.926731 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:44.928171 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 98d60f2f165a45a59bc9e98e907baf8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=278c402102934f588cc94055d6ba9c07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:44.928149 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 98d60f2f165a45a59bc9e98e907baf8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=278c402102934f588cc94055d6ba9c07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:44.928203 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 98d60f2f165a45a59bc9e98e907baf8e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=278c402102934f588cc94055d6ba9c07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:44.928334 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d60f2f165a45a59bc9e98e907baf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.928333 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d60f2f165a45a59bc9e98e907baf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.928387 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d60f2f165a45a59bc9e98e907baf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:44.929540 2018 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: Bootstrap starting.
I20260928 17:51:44.929785 2016 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: Bootstrap starting.
I20260928 17:51:44.929925 2018 tablet_bootstrap.cc:654] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.930187 2016 tablet_bootstrap.cc:654] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.930567 2018 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: No bootstrap required, opened a new log
I20260928 17:51:44.930567 2015 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: Bootstrap starting.
I20260928 17:51:44.930608 2018 ts_tablet_manager.cc:1397] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.930732 2018 raft_consensus.cc:348] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.930805 2018 raft_consensus.cc:374] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.930825 2018 raft_consensus.cc:729] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c5db8845544cad9954688b9c4a3df9, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.930850 2016 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: No bootstrap required, opened a new log
I20260928 17:51:44.930879 2018 consensus_queue.cc:260] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.930913 2016 ts_tablet_manager.cc:1397] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:44.930934 2015 tablet_bootstrap.cc:654] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:44.930987 2018 ts_tablet_manager.cc:1428] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:44.931020 2025 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.931072 2016 raft_consensus.cc:348] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.931110 2025 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.931133 2016 raft_consensus.cc:374] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.931180 2016 raft_consensus.cc:729] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1c969cac484ee7970179b2bf5c9d47, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.931228 2016 consensus_queue.cc:260] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.931339 2016 ts_tablet_manager.cc:1428] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.931576 2045 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.931661 2045 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.932328 2015 tablet_bootstrap.cc:492] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: No bootstrap required, opened a new log
I20260928 17:51:44.932371 2015 ts_tablet_manager.cc:1397] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:51:44.932513 2015 raft_consensus.cc:348] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.932578 2015 raft_consensus.cc:374] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:44.932592 2015 raft_consensus.cc:729] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea262c58010e409fa4ec3439bd838288, State: Initialized, Role: FOLLOWER
I20260928 17:51:44.932655 2015 consensus_queue.cc:260] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.932765 2015 ts_tablet_manager.cc:1428] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:44.932842 2023 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.932955 2023 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.968484 2045 raft_consensus.cc:482] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:44.968601 2045 raft_consensus.cc:504] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.968813 2045 leader_election.cc:288] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371), ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867)
I20260928 17:51:44.969038 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d60f2f165a45a59bc9e98e907baf8e" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288" is_pre_election: true
I20260928 17:51:44.969033 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d60f2f165a45a59bc9e98e907baf8e" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5db8845544cad9954688b9c4a3df9" is_pre_election: true
I20260928 17:51:44.969131 1806 raft_consensus.cc:2454] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 0.
I20260928 17:51:44.969131 1666 raft_consensus.cc:2454] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 0.
I20260928 17:51:44.969300 1878 leader_election.cc:302] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5db8845544cad9954688b9c4a3df9, ec1c969cac484ee7970179b2bf5c9d47; no voters:
I20260928 17:51:44.969409 2045 raft_consensus.cc:2781] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:44.969448 2045 raft_consensus.cc:482] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:44.969468 2045 raft_consensus.cc:3037] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.970199 2045 raft_consensus.cc:504] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.970350 2045 leader_election.cc:288] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 election: Requested vote from peers b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371), ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867)
I20260928 17:51:44.970520 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d60f2f165a45a59bc9e98e907baf8e" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c5db8845544cad9954688b9c4a3df9"
I20260928 17:51:44.970597 1666 raft_consensus.cc:3037] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.970525 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d60f2f165a45a59bc9e98e907baf8e" candidate_uuid: "ec1c969cac484ee7970179b2bf5c9d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea262c58010e409fa4ec3439bd838288"
I20260928 17:51:44.970646 1806 raft_consensus.cc:3037] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:44.971212 1666 raft_consensus.cc:2454] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 1.
I20260928 17:51:44.971369 1806 raft_consensus.cc:2454] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1c969cac484ee7970179b2bf5c9d47 in term 1.
I20260928 17:51:44.971376 1878 leader_election.cc:302] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c5db8845544cad9954688b9c4a3df9, ec1c969cac484ee7970179b2bf5c9d47; no voters:
I20260928 17:51:44.971486 2045 raft_consensus.cc:2781] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:44.971539 2045 raft_consensus.cc:686] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [term 1 LEADER]: Becoming Leader. State: Replica: ec1c969cac484ee7970179b2bf5c9d47, State: Running, Role: LEADER
I20260928 17:51:44.971590 2045 consensus_queue.cc:237] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } }
I20260928 17:51:44.972028 2021 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: TxnStatusTablet state changed. Reason: New leader ec1c969cac484ee7970179b2bf5c9d47. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.972154 2021 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:44.972340 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:51:44.972388 1508 catalog_manager.cc:5495] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } health_report { overall_health: HEALTHY } } }
I20260928 17:51:44.972498 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:51:44.979336 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.976587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.0.98.3:44003)
0928 17:51:44.976629 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.976631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.976657 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:51:44.976749 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.976750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.976763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.976819 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:44.976984 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.977501 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.978615 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.978794 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.978846 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.979054 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.979136 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.979187 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.979271 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.979287 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:51:44.979396 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.976532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44003 (local address 127.0.0.1:45370)
0928 17:51:44.976610 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.976621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.976674 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.976825 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.976826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.976827 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:44.976879 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.976883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.977584 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.977585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.978537 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.978540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.978872 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.978874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.978948 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.978964 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.979202 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.979210 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.979267 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.979321 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 17:51:44.980832 1806 raft_consensus.cc:1261] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Refusing update from remote peer ec1c969cac484ee7970179b2bf5c9d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.980967 1666 raft_consensus.cc:1261] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Refusing update from remote peer ec1c969cac484ee7970179b2bf5c9d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.981078 2045 consensus_queue.cc:1035] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:44.981232 2045 consensus_queue.cc:1035] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:44.981725 2023 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: TxnStatusTablet state changed. Reason: New leader ec1c969cac484ee7970179b2bf5c9d47. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.981839 2023 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.982299 2025 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: TxnStatusTablet state changed. Reason: New leader ec1c969cac484ee7970179b2bf5c9d47. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.982396 2025 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.982609 2021 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.982684 2021 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:44.982853 2021 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.982940 2021 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ec1c969cac484ee7970179b2bf5c9d47: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:44.983232 2023 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.983314 2023 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P ea262c58010e409fa4ec3439bd838288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.984054 2025 tablet_replica.cc:439] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec1c969cac484ee7970179b2bf5c9d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 } } peers { permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "ec1c969cac484ee7970179b2bf5c9d47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44003 } } }
I20260928 17:51:44.984455 2025 tablet_replica.cc:442] T 98d60f2f165a45a59bc9e98e907baf8e P b3c5db8845544cad9954688b9c4a3df9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:44.986919 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.984592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.0.98.3:44003)
0928 17:51:44.984622 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.984623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.984647 (+ 24us) server_negotiation.cc:407] Connection header received
0928 17:51:44.984721 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.984721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.984729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.984774 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:44.984913 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.985326 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.986354 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.986510 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.986575 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.986732 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.986754 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.986791 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.986845 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.986857 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 17:51:44.991677 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.989172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44003 (local address 127.0.98.3:60445)
0928 17:51:44.989235 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:44.989246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:44.989283 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:44.989403 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:44.989404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:44.989405 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:44.989447 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.989448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.990257 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.990258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.990888 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:44.990890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.991330 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.991331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:44.991375 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.991386 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.991548 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.991549 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:51:44.991582 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 17:51:44.991609 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:51:44.991683 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:44.989249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60445 (local address 127.0.98.3:44003)
0928 17:51:44.989287 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:44.989289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:44.989319 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:51:44.989338 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:44.989338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:44.989347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:44.989397 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:44.989490 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.990189 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.990970 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:44.991259 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:44.991326 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:44.991462 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:44.991491 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:44.991543 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:44.991587 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 17:51:44.991602 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:51:44.993522 1666 raft_consensus.cc:1261] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Refusing update from remote peer ec1c969cac484ee7970179b2bf5c9d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.993518 1806 raft_consensus.cc:1261] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Refusing update from remote peer ec1c969cac484ee7970179b2bf5c9d47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:44.993739 2064 consensus_queue.cc:1035] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c5db8845544cad9954688b9c4a3df9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:44.993853 2045 consensus_queue.cc:1035] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea262c58010e409fa4ec3439bd838288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:45.004843 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:45.004946 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:45.005635 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62025f68ca44943a944c24a8c09f206: Sending DeleteTablet for 3 replicas of tablet e48f785bf5a74a2597df2c9a8ea996af
I20260928 17:51:45.005898 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f785bf5a74a2597df2c9a8ea996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51286
I20260928 17:51:45.005916 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f785bf5a74a2597df2c9a8ea996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55710
I20260928 17:51:45.005955 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet e48f785bf5a74a2597df2c9a8ea996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45364
I20260928 17:51:45.005990 2040 tablet_replica.cc:330] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: stopping tablet replica
I20260928 17:51:45.006006 2039 tablet_replica.cc:330] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: stopping tablet replica
I20260928 17:51:45.006040 2040 raft_consensus.cc:2229] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:45.006047 2039 raft_consensus.cc:2229] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:45.006076 2040 raft_consensus.cc:2258] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:45.006079 2039 raft_consensus.cc:2258] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:45.006132 2038 tablet_replica.cc:330] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: stopping tablet replica
I20260928 17:51:45.006184 2038 raft_consensus.cc:2229] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:45.006295 2038 raft_consensus.cc:2258] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:45.006426 2039 ts_tablet_manager.cc:1905] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:45.007845 2038 ts_tablet_manager.cc:1905] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:45.007899 2039 ts_tablet_manager.cc:1918] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:45.007958 2039 log.cc:1196] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/e48f785bf5a74a2597df2c9a8ea996af
I20260928 17:51:45.008219 2039 ts_tablet_manager.cc:1939] T e48f785bf5a74a2597df2c9a8ea996af P ea262c58010e409fa4ec3439bd838288: Deleting consensus metadata
I20260928 17:51:45.008558 1496 catalog_manager.cc:4841] TS ea262c58010e409fa4ec3439bd838288 (127.0.98.2:35867): tablet e48f785bf5a74a2597df2c9a8ea996af (table test-table [id=90ece48c39e14809ad9765b3e2808f92]) successfully deleted
I20260928 17:51:45.009474 2038 ts_tablet_manager.cc:1918] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:45.009526 2038 log.cc:1196] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/e48f785bf5a74a2597df2c9a8ea996af
I20260928 17:51:45.009719 2038 ts_tablet_manager.cc:1939] T e48f785bf5a74a2597df2c9a8ea996af P ec1c969cac484ee7970179b2bf5c9d47: Deleting consensus metadata
I20260928 17:51:45.010006 1494 catalog_manager.cc:4841] TS ec1c969cac484ee7970179b2bf5c9d47 (127.0.98.3:44003): tablet e48f785bf5a74a2597df2c9a8ea996af (table test-table [id=90ece48c39e14809ad9765b3e2808f92]) successfully deleted
I20260928 17:51:45.010058 2040 ts_tablet_manager.cc:1905] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:45.011340 2040 ts_tablet_manager.cc:1918] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:45.011395 2040 log.cc:1196] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/e48f785bf5a74a2597df2c9a8ea996af
I20260928 17:51:45.011600 2040 ts_tablet_manager.cc:1939] T e48f785bf5a74a2597df2c9a8ea996af P b3c5db8845544cad9954688b9c4a3df9: Deleting consensus metadata
I20260928 17:51:45.011900 1494 catalog_manager.cc:4841] TS b3c5db8845544cad9954688b9c4a3df9 (127.0.98.1:41371): tablet e48f785bf5a74a2597df2c9a8ea996af (table test-table [id=90ece48c39e14809ad9765b3e2808f92]) successfully deleted
Sep 28 17:51:45 dist-test-slave-zqlt krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:45.017127 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:45.006662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.0.98.1:41371)
0928 17:51:45.006699 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:45.006701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:45.006833 (+ 132us) server_negotiation.cc:407] Connection header received
0928 17:51:45.006927 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:45.006928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:45.006938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:45.006968 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:45.008929 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:45.009667 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:45.010737 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:45.010990 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:45.011060 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:45.014211 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:45.014219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:45.014220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:45.014228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:45.015868 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:45.016313 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:45.016317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:45.016317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:45.016378 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:45.016577 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:45.016578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:45.016578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:45.016726 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:45.016891 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:45.016938 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 17:51:45.017015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:51:45.017376 1706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306
I20260928 17:51:45.024703 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:45.018364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.98.62:44799)
0928 17:51:45.018409 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:45.018411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:45.018528 (+ 117us) server_negotiation.cc:407] Connection header received
0928 17:51:45.018601 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:45.018601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:45.018612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:45.018657 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:45.019142 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:45.019645 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:45.020445 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:45.020745 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:45.020834 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:45.022100 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:45.022102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:45.022102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:45.022110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:45.023582 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:45.023927 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:45.023928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:45.023929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:45.023993 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:45.024201 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:45.024202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:45.024202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:45.024321 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:45.024423 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:45.024518 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:51:45.024598 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 17:51:45.024945 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38826
I20260928 17:51:45.025606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1581
I20260928 17:51:45.032822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1721
I20260928 17:51:45.037878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1861
I20260928 17:51:45.042908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 1477
2026-09-28T17:51:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2318 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:45 dist-test-slave-zqlt krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:45 dist-test-slave-zqlt krb5kdc[2074](info): set up 2 sockets
Sep 28 17:51:45 dist-test-slave-zqlt krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260928 17:51:46.079761 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:46.193240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:46.273089 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:46.273313 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:46.273336 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:46.274469 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:46.274509 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:51:46.274523 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:46.274536 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:46.274549 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:46.274561 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:46.275985 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:46.276311 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:46.276544 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:51:46.281648 2087 server_base.cc:1034] running on GCE node
W20260928 17:51:46.281622 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.281622 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.281618 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:46.282141 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:46.282442 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:46.283612 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790617906283595 us; error 30 us; skew 500 ppm
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:46.286387 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:46.286890 2087 webserver.cc:460] Webserver started at http://127.0.98.62:36949/ using document root <none> and password file <none>
I20260928 17:51:46.287127 2087 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:46.287173 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:46.287321 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:46.288537 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "213da64453dd44778342760ec1eef153"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "bfd06b2b26480df04f81759defe06d54"
server_key_iv: "5e243a76da6798507a212b97f7b38103"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.288914 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "213da64453dd44778342760ec1eef153"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "bfd06b2b26480df04f81759defe06d54"
server_key_iv: "5e243a76da6798507a212b97f7b38103"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.290355 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:46.291246 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.291445 2087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:46.291534 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "213da64453dd44778342760ec1eef153"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "bfd06b2b26480df04f81759defe06d54"
server_key_iv: "5e243a76da6798507a212b97f7b38103"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.291591 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:46.317759 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:46.320216 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:46.320338 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:46.324877 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37487
I20260928 17:51:46.324975 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37487 every 8 connection(s)
I20260928 17:51:46.325294 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:46.325989 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:46.327710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2087
I20260928 17:51:46.327783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:46.327956 392 external_mini_cluster.cc:1428] Setting key 95fa41010c6227da65ab5fb7c5ca477e
I20260928 17:51:46.328518 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Bootstrap starting.
I20260928 17:51:46.329269 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:46.329608 2165 log.cc:824] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:46.330372 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: No bootstrap required, opened a new log
I20260928 17:51:46.332746 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } }
I20260928 17:51:46.332978 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:46.333014 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213da64453dd44778342760ec1eef153, State: Initialized, Role: FOLLOWER
I20260928 17:51:46.333149 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } }
I20260928 17:51:46.333233 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:46.333263 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:46.333308 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:46.333994 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } }
I20260928 17:51:46.334129 2165 leader_election.cc:302] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213da64453dd44778342760ec1eef153; no voters:
I20260928 17:51:46.334441 2165 leader_election.cc:288] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:46.334467 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:46.334862 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [term 1 LEADER]: Becoming Leader. State: Replica: 213da64453dd44778342760ec1eef153, State: Running, Role: LEADER
I20260928 17:51:46.334959 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:46.335026 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } }
I20260928 17:51:46.335414 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 213da64453dd44778342760ec1eef153. Latest consensus state: current_term: 1 leader_uuid: "213da64453dd44778342760ec1eef153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } } }
I20260928 17:51:46.335423 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "213da64453dd44778342760ec1eef153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213da64453dd44778342760ec1eef153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37487 } } }
I20260928 17:51:46.335532 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:46.335544 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153 [sys.catalog]: This master's current role is: LEADER
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:46.336575 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:46.337661 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:46.339504 2177 catalog_manager.cc:1347] Generated new cluster ID: 18ac4c88fb6444eb9ae8863c44af2107
I20260928 17:51:46.339555 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:51:46.340502 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.328515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.0.98.62:37487)
0928 17:51:46.328715 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.328724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.328756 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:51:46.329424 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.329437 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.329516 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.329703 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.330469 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.331975 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.332817 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.333429 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.333565 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.336519 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.336523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.336524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.336538 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.338465 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.339062 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.339066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.339066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.339204 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.339428 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.339429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.339429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.339738 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.339854 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.340166 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.340309 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 17:51:46.350719 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:46.351263 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:46.359122 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Generated new TSK 0
I20260928 17:51:46.359364 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:46.390980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37487
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:46.494935 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:46.495162 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:46.495204 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:46.496544 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:46.496608 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:46.496650 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:51:46.496676 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:46.496711 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:46.498136 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:46.498391 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:46.498616 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:46.503532 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.503545 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.503554 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:46.503708 2191 server_base.cc:1034] running on GCE node
I20260928 17:51:46.503924 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:46.504169 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:46.505316 2191 hybrid_clock.cc:648] HybridClock initialized: now 1790617906505307 us; error 27 us; skew 500 ppm
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:46.507653 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:46.508131 2191 webserver.cc:460] Webserver started at http://127.0.98.1:33479/ using document root <none> and password file <none>
I20260928 17:51:46.508409 2191 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:46.508476 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:46.508628 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:46.509639 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbd9941751924fcfadd83192ed645b79"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "0fef86d37fc996739b0facba244a1d6a"
server_key_iv: "76d25308d9b3e6ff0d676a73d7bcf1fa"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.510000 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd9941751924fcfadd83192ed645b79"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "0fef86d37fc996739b0facba244a1d6a"
server_key_iv: "76d25308d9b3e6ff0d676a73d7bcf1fa"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.511271 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:51:46.511925 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.512074 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.512203 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "dbd9941751924fcfadd83192ed645b79"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "0fef86d37fc996739b0facba244a1d6a"
server_key_iv: "76d25308d9b3e6ff0d676a73d7bcf1fa"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.512286 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:46.523804 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:46.525652 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:46.526093 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:46.526681 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:46.526744 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.526782 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:46.526809 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.533916 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41421
I20260928 17:51:46.533977 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41421 every 8 connection(s)
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:46.534399 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:51:46.537540 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.527171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46655 (local address 127.0.98.62:37487)
0928 17:51:46.527220 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.527222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.527438 (+ 216us) server_negotiation.cc:407] Connection header received
0928 17:51:46.527966 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.527968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.527981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.528040 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.528969 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.529814 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.531442 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.531840 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.531917 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.534907 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.534909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.534909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.534919 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.536408 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.536832 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.536833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.536833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.536884 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.537061 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.537061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.537061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.537165 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.537210 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.537408 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.537476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:51:46.537736 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.527166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.1:46655)
0928 17:51:46.527315 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.527349 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.527860 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.528149 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.528152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.528536 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.528867 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.528874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.529905 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.529906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.531335 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.531338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.531924 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.531925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.532131 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.532767 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.532779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.534760 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.536484 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.536490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.536492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.536775 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.536928 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.536928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.536929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.537014 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.537202 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.537204 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.537320 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.537397 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.537521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260928 17:51:46.538260 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37487
I20260928 17:51:46.538348 2322 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:46.538537 2322 heartbeater.cc:511] Master 127.0.98.62:37487 requested a full tablet report, sending...
I20260928 17:51:46.538635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2191
I20260928 17:51:46.538735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:46.538928 392 external_mini_cluster.cc:1428] Setting key 25c5acf955e3bc59b12586900e603740
I20260928 17:51:46.539841 2119 ts_manager.cc:194] Registered new tserver with Master: dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421)
I20260928 17:51:46.540611 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46655
I20260928 17:51:46.545794 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.539348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39085 (local address 127.0.98.62:37487)
0928 17:51:46.540161 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.540163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.540171 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:51:46.540190 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.540191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.540200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.540262 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.540654 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.541343 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.542129 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.542316 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.542371 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.543380 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.543381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.543382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.543387 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.544712 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.545118 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.545119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.545119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.545164 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.545375 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.545376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.545376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.545464 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.545511 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.545679 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.545738 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792}
I20260928 17:51:46.545884 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.539304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.1:39085)
0928 17:51:46.539443 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.539452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.539502 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.540283 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.540284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.540501 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.540573 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.540574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.541451 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.541452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.542062 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.542064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.542391 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.542392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.542475 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.542876 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.542881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.543293 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.544793 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.544794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.544795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.545039 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.545241 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.545242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.545242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.545313 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.545526 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.545527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.545581 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.545657 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.545772 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:46.564513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37487
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:46.645987 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:46.646193 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:46.646217 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:46.647380 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:46.647423 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:46.647444 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:51:46.647495 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:46.647526 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:46.649128 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:46.649374 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:46.649600 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:46.654711 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.654716 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.654826 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:46.655136 2331 server_base.cc:1034] running on GCE node
I20260928 17:51:46.655320 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:46.655517 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:46.656661 2331 hybrid_clock.cc:648] HybridClock initialized: now 1790617906656647 us; error 28 us; skew 500 ppm
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:46.659132 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:46.659513 2331 webserver.cc:460] Webserver started at http://127.0.98.2:39661/ using document root <none> and password file <none>
I20260928 17:51:46.659699 2331 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:46.659740 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:46.659824 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:46.660856 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "8548eb776c694e55a4bca85fdfae389f"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "3956618494a341dd420148023116dd0e"
server_key_iv: "f67ec5de2c16a0b346fbaa4be5eaf1e0"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.661145 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8548eb776c694e55a4bca85fdfae389f"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "3956618494a341dd420148023116dd0e"
server_key_iv: "f67ec5de2c16a0b346fbaa4be5eaf1e0"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.662330 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:51:46.662940 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.663095 2331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:46.663189 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "8548eb776c694e55a4bca85fdfae389f"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "3956618494a341dd420148023116dd0e"
server_key_iv: "f67ec5de2c16a0b346fbaa4be5eaf1e0"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.663267 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:46.671960 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:46.674269 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:46.674425 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:46.674885 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:46.674943 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.674983 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:46.675011 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:46.681859 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36131
I20260928 17:51:46.681924 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36131 every 8 connection(s)
I20260928 17:51:46.682262 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:46.684758 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.675516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36605 (local address 127.0.98.62:37487)
0928 17:51:46.675566 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.675568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.675709 (+ 141us) server_negotiation.cc:407] Connection header received
0928 17:51:46.676290 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.676292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.676305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.676369 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.677066 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.677584 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.679166 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.679504 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.679570 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.682247 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.682249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.682249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.682258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.683536 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.683910 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.683911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.683912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.683962 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.684200 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.684201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.684202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.684316 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.684370 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.684616 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.684695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:51:46.684918 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.675396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.2:36605)
0928 17:51:46.675568 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.675617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.676172 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.676460 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.676466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.676783 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.676964 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.676969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.677679 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.677680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.679087 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.679090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.679599 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.679600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.679748 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.680657 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.680665 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.682148 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.683611 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.683616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.683618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.683841 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.684003 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.684003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.684004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.684133 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.684374 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.684378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.684494 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.684571 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.684704 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260928 17:51:46.685451 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37487
I20260928 17:51:46.685532 2462 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:46.685719 2462 heartbeater.cc:511] Master 127.0.98.62:37487 requested a full tablet report, sending...
I20260928 17:51:46.686084 2119 ts_manager.cc:194] Registered new tserver with Master: 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:46.686596 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36605
I20260928 17:51:46.689401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2331
I20260928 17:51:46.689493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:46.689705 392 external_mini_cluster.cc:1428] Setting key 137c4baebe896bf7682b62281b3cf724
I20260928 17:51:46.692435 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.686582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56833 (local address 127.0.98.62:37487)
0928 17:51:46.686623 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.686625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.686732 (+ 107us) server_negotiation.cc:407] Connection header received
0928 17:51:46.686805 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.686806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.686815 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.686898 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.687214 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.687757 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.688487 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.688662 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.688724 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.689665 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.689666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.689666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.689673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.691198 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.691579 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.691580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.691581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.691644 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.691848 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.691848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.691849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.692000 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.692074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.692259 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.692339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:51:46.692449 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.686524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.2:56833)
0928 17:51:46.686673 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.686685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.686754 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.686915 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.686916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.687075 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.687129 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.687131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.687840 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.687841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.688421 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.688422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.688734 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.688735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.688793 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.689169 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.689173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.689585 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.691308 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.691309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.691309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.691490 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.691715 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.691716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.691716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.691775 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.692076 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.692077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.692143 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.692231 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.692340 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:46.714881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37487
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:46.796401 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:46.796592 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:46.796615 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:46.797690 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:46.797724 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:46.797742 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 17:51:46.797755 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:51:46.797771 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:46.799090 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:46.799312 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:46.799501 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:46.804157 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.804198 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:46.804209 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:46.804486 2471 server_base.cc:1034] running on GCE node
I20260928 17:51:46.804711 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:46.804950 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:46.806107 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790617906806097 us; error 53 us; skew 500 ppm
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:46.808501 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:46.808923 2471 webserver.cc:460] Webserver started at http://127.0.98.3:33881/ using document root <none> and password file <none>
I20260928 17:51:46.809142 2471 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:46.809211 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:46.809361 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:46.810377 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "a155fb2dbd1f481ea24972ad03b35a45"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "7854257e9f39f3e4e3881898fdec2142"
server_key_iv: "b8e605e9ffeebf23482638c8c256456b"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.810743 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a155fb2dbd1f481ea24972ad03b35a45"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "7854257e9f39f3e4e3881898fdec2142"
server_key_iv: "b8e605e9ffeebf23482638c8c256456b"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.812027 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:51:46.812764 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.812938 2471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:46.813037 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "a155fb2dbd1f481ea24972ad03b35a45"
format_stamp: "Formatted at 2026-09-28 17:51:46 on dist-test-slave-zqlt"
server_key: "7854257e9f39f3e4e3881898fdec2142"
server_key_iv: "b8e605e9ffeebf23482638c8c256456b"
server_key_version: "encryptionkey@0"
I20260928 17:51:46.813122 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:46.834090 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:46.836500 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:46.836684 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:46.837226 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:46.837293 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.837344 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:46.837369 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:46.844683 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33039
I20260928 17:51:46.844722 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33039 every 8 connection(s)
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:46.845144 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:46.847934 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.837779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52555 (local address 127.0.98.62:37487)
0928 17:51:46.837852 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.837855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.838091 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:51:46.838699 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.838701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.838716 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.838794 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.839766 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.840681 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.842476 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.842781 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.842839 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.845620 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.845621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.845622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.845629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.846792 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.847156 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.847157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.847157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.847216 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.847422 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.847423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.847423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.847526 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.847575 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.847809 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.847870 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:51:46.848181 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.837773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.3:52555)
0928 17:51:46.837949 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.837991 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.838570 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.838863 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.838884 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.839295 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.839655 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.839662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.840809 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.840810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.842355 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.842358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.842873 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.842874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.843046 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.843684 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.843695 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.845528 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.846860 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.846866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.846867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.847088 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.847273 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.847274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.847274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.847368 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.847579 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.847583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.847696 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.847780 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.847904 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260928 17:51:46.848694 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37487
I20260928 17:51:46.848783 2602 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:46.848981 2602 heartbeater.cc:511] Master 127.0.98.62:37487 requested a full tablet report, sending...
I20260928 17:51:46.849342 2119 ts_manager.cc:194] Registered new tserver with Master: a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039)
I20260928 17:51:46.849838 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52555
I20260928 17:51:46.850049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2471
I20260928 17:51:46.850136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:46.850319 392 external_mini_cluster.cc:1428] Setting key 527e0f54b513d9cec9a232b2d7c60b68
I20260928 17:51:46.851090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:51:46.856472 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.849721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48009 (local address 127.0.98.62:37487)
0928 17:51:46.849761 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.849763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.849825 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:51:46.850457 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.850458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.850468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.850526 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.850961 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.851652 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.852773 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.852950 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.853042 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.853984 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.853985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.853985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.853992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.855337 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.855683 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.855684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.855685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.855742 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.855942 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.855943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.855944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.856052 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.856149 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.856304 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.856382 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:51:46.856520 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.849625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.98.3:48009)
0928 17:51:46.849763 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.849773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.849843 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.850567 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.850568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.850783 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:46.850843 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.850844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.851794 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.851795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.852698 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.852701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.853044 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.853045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.853123 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.853590 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:46.853595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:46.853879 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:46.855428 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.855429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.855429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.855597 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.855806 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:46.855807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:46.855807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:46.855864 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:46.856138 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:46.856139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:46.856191 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.856278 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.856385 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:51:46 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:46.875514 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.867551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.0.98.62:37487)
0928 17:51:46.867590 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.867592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.867601 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:51:46.867686 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.867687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.867695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.867743 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:46.868571 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.869102 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.870263 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.870440 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.870489 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.873053 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:46.873055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:46.873055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:46.873063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:46.874409 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.874750 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.874751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.874752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.874806 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:46.875021 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:46.875022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:46.875023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:46.875120 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:46.875231 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.875331 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.875409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:51:46.877159 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:46.878661 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:46.885155 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.880830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.0.1:50412)
0928 17:51:46.880979 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.881019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.881083 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.881503 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.881507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.881509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:46.881609 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.881611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.882932 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.882933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.883964 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.883966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.884871 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.884962 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.884969 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.885002 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.885045 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":29,"thread_start_us":50,"threads_started":1}
I20260928 17:51:46.885140 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.881201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.98.3:33039)
0928 17:51:46.881358 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.881363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.881379 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:51:46.881409 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.881411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.881422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.881762 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:46.881822 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.882505 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.883909 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.884700 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.884850 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.884986 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.885033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 17:51:46.885102 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.881008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33039 (local address 127.0.0.1:53058)
0928 17:51:46.881168 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.881187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.881227 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.881541 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.881542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.881544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:46.881674 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.881676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.882594 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.882595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.883837 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.883840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.884791 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.884876 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.884891 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.884961 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.885012 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260928 17:51:46.885461 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.881011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.98.1:41421)
0928 17:51:46.881310 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.881316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.881326 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:51:46.881369 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.881371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.881386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.881474 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:46.881699 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.882850 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884038 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.884778 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.885146 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.885224 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.885320 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.885370 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":78,"threads_started":1}
I20260928 17:51:46.885962 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 (DEFAULT_TABLE table=test-table [id=67e81f3edde54a428a671a778ef3fd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:46.886384 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9b5318621d44618cc1d1be236b4ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:46.886456 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.880830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36131 (local address 127.0.0.1:39614)
0928 17:51:46.881119 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:46.881130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:46.881190 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:46.881506 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:46.881509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:46.881510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:46.881586 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.881587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.884031 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.884786 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:46.884788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.886142 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.886143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:46.886237 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.886241 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:46.886300 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:51:46.886387 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260928 17:51:46.886466 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:46.880922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.0.98.2:36131)
0928 17:51:46.881339 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:46.881343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:46.881351 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:51:46.881385 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:46.881388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:46.881401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:46.881666 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:46.881725 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.882843 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.884882 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:46.886000 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:46.886081 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:46.886152 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:46.886317 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 17:51:46.886378 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":70,"threads_started":1}
I20260928 17:51:46.887079 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 (DEFAULT_TABLE table=test-table [id=67e81f3edde54a428a671a778ef3fd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:46.887354 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9b5318621d44618cc1d1be236b4ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:46.889312 2625 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Bootstrap starting.
I20260928 17:51:46.889765 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 (DEFAULT_TABLE table=test-table [id=67e81f3edde54a428a671a778ef3fd17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:46.889997 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9b5318621d44618cc1d1be236b4ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:46.890316 2625 tablet_bootstrap.cc:654] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:46.890535 2626 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Bootstrap starting.
I20260928 17:51:46.890640 2625 log.cc:824] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:46.891144 2626 tablet_bootstrap.cc:654] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:46.891325 2625 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: No bootstrap required, opened a new log
I20260928 17:51:46.891371 2625 ts_tablet_manager.cc:1397] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:46.891470 2626 log.cc:824] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:46.892697 2626 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: No bootstrap required, opened a new log
I20260928 17:51:46.892789 2626 ts_tablet_manager.cc:1397] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:46.893239 2625 raft_consensus.cc:348] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.893390 2625 raft_consensus.cc:374] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:46.893448 2625 raft_consensus.cc:729] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Initialized, Role: FOLLOWER
I20260928 17:51:46.893592 2625 consensus_queue.cc:260] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.893774 2629 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Bootstrap starting.
I20260928 17:51:46.893877 2625 ts_tablet_manager.cc:1428] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:51:46.893903 2602 heartbeater.cc:503] Master 127.0.98.62:37487 was elected leader, sending a full tablet report...
I20260928 17:51:46.894856 2629 tablet_bootstrap.cc:654] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:46.894915 2626 raft_consensus.cc:348] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.895083 2626 raft_consensus.cc:374] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:46.895136 2626 raft_consensus.cc:729] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Initialized, Role: FOLLOWER
I20260928 17:51:46.895283 2629 log.cc:824] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:46.895257 2626 consensus_queue.cc:260] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.895557 2626 ts_tablet_manager.cc:1428] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:51:46.895570 2462 heartbeater.cc:503] Master 127.0.98.62:37487 was elected leader, sending a full tablet report...
I20260928 17:51:46.896085 2629 tablet_bootstrap.cc:492] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: No bootstrap required, opened a new log
I20260928 17:51:46.896179 2629 ts_tablet_manager.cc:1397] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:51:46.897601 2629 raft_consensus.cc:348] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.897722 2629 raft_consensus.cc:374] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:46.897744 2629 raft_consensus.cc:729] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Initialized, Role: FOLLOWER
I20260928 17:51:46.897815 2629 consensus_queue.cc:260] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:46.897993 2629 ts_tablet_manager.cc:1428] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:51:46.898062 2322 heartbeater.cc:503] Master 127.0.98.62:37487 was elected leader, sending a full tablet report...
W20260928 17:51:46.933066 2463 tablet.cc:2369] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:51:47.035300 2323 tablet.cc:2369] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:51:47.095767 2603 tablet.cc:2369] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:51:47.165678 2630 raft_consensus.cc:482] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:47.165863 2630 raft_consensus.cc:504] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:47.166255 2630 leader_election.cc:288] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:47.169552 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.166526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51601 (local address 127.0.98.2:36131)
0928 17:51:47.166581 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.166583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.166596 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:51:47.166697 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.166698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.166716 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.166776 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:47.166973 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.167467 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.168807 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.169189 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.169276 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.169288 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.169442 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.169492 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:51:47.169550 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.166454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36131 (local address 127.0.98.3:51601)
0928 17:51:47.166511 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.166524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.166570 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.166804 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.166806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.166808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:47.166887 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.166889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.167539 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.167540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.168713 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.168715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.169273 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.169274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.169353 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.169356 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.169432 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.169472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:51:47.169965 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9b5318621d44618cc1d1be236b4ee5" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f" is_pre_election: true
I20260928 17:51:47.170158 2416 raft_consensus.cc:2454] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:47.170193 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.166468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.98.3:47853)
0928 17:51:47.166701 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.166710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.166764 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.167561 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.167561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.167563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:47.167623 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.167624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.168529 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.168530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.169246 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.169248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.169908 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.169910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.170004 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.170008 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.170066 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.170105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260928 17:51:47.170244 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.166604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47853 (local address 127.0.98.1:41421)
0928 17:51:47.166646 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.166648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.166833 (+ 185us) server_negotiation.cc:407] Connection header received
0928 17:51:47.166863 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.166864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.166896 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.166950 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:47.167689 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.168448 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.169348 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.169820 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.169883 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.169891 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.170078 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.170143 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:51:47.170397 2490 leader_election.cc:302] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8548eb776c694e55a4bca85fdfae389f, a155fb2dbd1f481ea24972ad03b35a45; no voters:
I20260928 17:51:47.170598 2630 raft_consensus.cc:2781] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:47.170653 2630 raft_consensus.cc:482] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:47.170671 2630 raft_consensus.cc:3037] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.170609 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9b5318621d44618cc1d1be236b4ee5" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79" is_pre_election: true
I20260928 17:51:47.170743 2276 raft_consensus.cc:2454] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:47.171729 2630 raft_consensus.cc:504] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:47.171885 2630 leader_election.cc:288] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Requested vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:47.172060 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9b5318621d44618cc1d1be236b4ee5" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79"
I20260928 17:51:47.172168 2276 raft_consensus.cc:3037] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.172065 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9b5318621d44618cc1d1be236b4ee5" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f"
I20260928 17:51:47.172272 2416 raft_consensus.cc:3037] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.172889 2276 raft_consensus.cc:2454] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:47.173065 2489 leader_election.cc:302] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:47.173101 2416 raft_consensus.cc:2454] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:47.173173 2630 raft_consensus.cc:2781] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:47.173327 2630 raft_consensus.cc:686] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Becoming Leader. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Running, Role: LEADER
I20260928 17:51:47.173426 2630 consensus_queue.cc:237] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } }
I20260928 17:51:47.174247 2119 catalog_manager.cc:5495] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a155fb2dbd1f481ea24972ad03b35a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } health_report { overall_health: UNKNOWN } } }
Sep 28 17:51:47 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:47.256213 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.249009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37487 (local address 127.0.0.1:52182)
0928 17:51:47.249168 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.249188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.249251 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.249488 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.249489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.249816 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:47.249865 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.249867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.250523 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.250523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.251052 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.251054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.251419 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.251421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.251472 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.252195 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:47.252199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:47.253576 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:47.255169 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:47.255176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:47.255178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:47.255377 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:47.255576 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:47.255577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:47.255577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:47.255636 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:47.255868 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:47.255873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:47.255915 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.255968 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.256066 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260928 17:51:47.256330 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.249118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.0.98.62:37487)
0928 17:51:47.249182 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.249184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.249286 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:51:47.249322 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.249323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.249334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.249458 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:47.249966 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.250446 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.251124 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.251309 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.251360 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.253675 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:47.253677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:47.253677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:47.253684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:47.255087 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:47.255450 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:47.255452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:47.255452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:47.255514 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:47.255692 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:47.255693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:47.255693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:47.255811 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:47.255868 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.256004 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.256191 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":36}
I20260928 17:51:47.258383 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:51:47.258620 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:47.260211 2396 tablet_service.cc:1467] Processing CreateTablet for tablet e09dc8d0a804490fa3697567612629b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3acff9cadc9c40aa81f03d188a8150fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:47.260211 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e09dc8d0a804490fa3697567612629b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3acff9cadc9c40aa81f03d188a8150fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:47.260388 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09dc8d0a804490fa3697567612629b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.260388 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09dc8d0a804490fa3697567612629b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.261909 2629 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: Bootstrap starting.
I20260928 17:51:47.261909 2626 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: Bootstrap starting.
I20260928 17:51:47.262027 2533 tablet_service.cc:1467] Processing CreateTablet for tablet e09dc8d0a804490fa3697567612629b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3acff9cadc9c40aa81f03d188a8150fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:51:47.262214 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09dc8d0a804490fa3697567612629b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.262358 2626 tablet_bootstrap.cc:654] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.262367 2629 tablet_bootstrap.cc:654] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.263032 2626 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: No bootstrap required, opened a new log
I20260928 17:51:47.263098 2626 ts_tablet_manager.cc:1397] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:47.263237 2629 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: No bootstrap required, opened a new log
I20260928 17:51:47.263274 2629 ts_tablet_manager.cc:1397] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:47.263295 2626 raft_consensus.cc:348] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.263374 2626 raft_consensus.cc:374] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.263394 2626 raft_consensus.cc:729] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.263444 2626 consensus_queue.cc:260] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.263550 2626 ts_tablet_manager.cc:1428] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:47.263603 2631 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.263684 2625 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: Bootstrap starting.
I20260928 17:51:47.263737 2631 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.263444 2629 raft_consensus.cc:348] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.263906 2629 raft_consensus.cc:374] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.263983 2629 raft_consensus.cc:729] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.264117 2629 consensus_queue.cc:260] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.264328 2625 tablet_bootstrap.cc:654] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.264330 2633 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.264408 2633 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.264576 2629 ts_tablet_manager.cc:1428] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:47.265054 2625 tablet_bootstrap.cc:492] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: No bootstrap required, opened a new log
I20260928 17:51:47.265110 2625 ts_tablet_manager.cc:1397] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:47.265252 2625 raft_consensus.cc:348] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.265334 2625 raft_consensus.cc:374] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.265349 2625 raft_consensus.cc:729] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.265412 2625 consensus_queue.cc:260] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.265511 2625 ts_tablet_manager.cc:1428] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:47.265646 2630 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.265760 2630 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.403751 2633 raft_consensus.cc:482] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:47.403846 2633 raft_consensus.cc:504] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.404215 2633 leader_election.cc:288] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131), a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039)
I20260928 17:51:47.407755 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.404489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59827 (local address 127.0.98.3:33039)
0928 17:51:47.404591 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.404597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.404606 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:51:47.404710 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.404711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.404722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.404814 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:47.405024 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.405778 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.406784 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.407344 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.407400 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.407409 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.407646 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.407689 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":60,"threads_started":1}
I20260928 17:51:47.407796 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.404427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33039 (local address 127.0.98.1:59827)
0928 17:51:47.404549 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.404561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.404623 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.404829 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.404830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.404833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:47.404940 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.404942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.405856 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.405858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.406679 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.406682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.407442 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.407443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.407557 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.407562 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.407643 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.407693 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":32,"thread_start_us":56,"threads_started":1}
I20260928 17:51:47.408138 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.404489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35759 (local address 127.0.98.2:36131)
0928 17:51:47.404537 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.404540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.404631 (+ 91us) server_negotiation.cc:407] Connection header received
0928 17:51:47.404727 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.404729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.404739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.404813 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:47.405016 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.405765 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.407217 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.407746 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.407817 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.407828 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.407996 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.408038 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:51:47.408205 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.404427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36131 (local address 127.0.98.1:35759)
0928 17:51:47.404582 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.404591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.404656 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.404829 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.404830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.404832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:47.404940 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.404943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.405891 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.405892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.407110 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.407113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.407829 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.407831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.407915 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.407921 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.407986 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.408032 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":23,"spinlock_wait_cycles":17280,"thread_start_us":57,"threads_started":1}
I20260928 17:51:47.408232 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09dc8d0a804490fa3697567612629b5" candidate_uuid: "dbd9941751924fcfadd83192ed645b79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a155fb2dbd1f481ea24972ad03b35a45" is_pre_election: true
I20260928 17:51:47.408376 2556 raft_consensus.cc:2454] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd9941751924fcfadd83192ed645b79 in term 0.
I20260928 17:51:47.408468 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09dc8d0a804490fa3697567612629b5" candidate_uuid: "dbd9941751924fcfadd83192ed645b79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f" is_pre_election: true
I20260928 17:51:47.408558 2416 raft_consensus.cc:2454] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd9941751924fcfadd83192ed645b79 in term 0.
I20260928 17:51:47.408581 2209 leader_election.cc:302] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:47.408730 2633 raft_consensus.cc:2781] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:47.408793 2633 raft_consensus.cc:482] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:47.408826 2633 raft_consensus.cc:3037] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.409969 2633 raft_consensus.cc:504] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.410147 2633 leader_election.cc:288] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [CANDIDATE]: Term 1 election: Requested vote from peers 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131), a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039)
I20260928 17:51:47.410277 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09dc8d0a804490fa3697567612629b5" candidate_uuid: "dbd9941751924fcfadd83192ed645b79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f"
I20260928 17:51:47.410305 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09dc8d0a804490fa3697567612629b5" candidate_uuid: "dbd9941751924fcfadd83192ed645b79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a155fb2dbd1f481ea24972ad03b35a45"
I20260928 17:51:47.410375 2416 raft_consensus.cc:3037] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.410377 2556 raft_consensus.cc:3037] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.411056 2556 raft_consensus.cc:2454] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd9941751924fcfadd83192ed645b79 in term 1.
I20260928 17:51:47.411072 2416 raft_consensus.cc:2454] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd9941751924fcfadd83192ed645b79 in term 1.
I20260928 17:51:47.411278 2209 leader_election.cc:302] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:47.411381 2633 raft_consensus.cc:2781] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:47.411494 2633 raft_consensus.cc:686] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [term 1 LEADER]: Becoming Leader. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Running, Role: LEADER
I20260928 17:51:47.411630 2633 consensus_queue.cc:237] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.412050 2649 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: TxnStatusTablet state changed. Reason: New leader dbd9941751924fcfadd83192ed645b79. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.412245 2649 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:47.412449 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:51:47.412582 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:51:47.412503 2119 catalog_manager.cc:5495] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbd9941751924fcfadd83192ed645b79 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } health_report { overall_health: UNKNOWN } } }
I20260928 17:51:47.446872 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.443951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.1:41421)
0928 17:51:47.443996 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.443998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.444032 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:51:47.444200 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.444202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.444219 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.444279 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:47.444438 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.444915 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.446051 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.446251 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.446307 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.446520 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.446624 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.446695 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.446769 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.446790 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:51:47.446905 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.443882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.0.1:50414)
0928 17:51:47.443975 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.443989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.444055 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.444309 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.444310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.444312 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:47.444368 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.444371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.444997 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.444999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.445954 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.445957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.446332 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.446333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.446405 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.446423 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.446694 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.446702 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.446764 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.446819 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 17:51:47.449103 2416 raft_consensus.cc:1261] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Refusing update from remote peer dbd9941751924fcfadd83192ed645b79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.449103 2556 raft_consensus.cc:1261] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Refusing update from remote peer dbd9941751924fcfadd83192ed645b79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.449468 2633 consensus_queue.cc:1035] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:47.449610 2649 consensus_queue.cc:1035] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:47.450737 2630 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: TxnStatusTablet state changed. Reason: New leader dbd9941751924fcfadd83192ed645b79. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.450838 2630 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.450798 2631 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: TxnStatusTablet state changed. Reason: New leader dbd9941751924fcfadd83192ed645b79. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.450888 2631 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.452301 2649 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.452394 2649 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:47.452438 2649 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.452476 2649 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P dbd9941751924fcfadd83192ed645b79: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:51:47.452627 2656 mvcc.cc:205] Tried to move back new op lower bound from 7334370948908445696 to 7334370948758614016. Current Snapshot: MvccSnapshot[applied={T|T < 7334370948908445696}]
I20260928 17:51:47.452597 2631 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.452728 2630 tablet_replica.cc:439] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd9941751924fcfadd83192ed645b79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } } }
I20260928 17:51:47.452783 2630 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P a155fb2dbd1f481ea24972ad03b35a45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.452883 2654 mvcc.cc:205] Tried to move back new op lower bound from 7334370948908445696 to 7334370948758614016. Current Snapshot: MvccSnapshot[applied={T|T < 7334370948908445696}]
I20260928 17:51:47.452701 2631 tablet_replica.cc:442] T e09dc8d0a804490fa3697567612629b5 P 8548eb776c694e55a4bca85fdfae389f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:51:47.452702 2655 mvcc.cc:205] Tried to move back new op lower bound from 7334370948908445696 to 7334370948758614016. Current Snapshot: MvccSnapshot[applied={T|T < 7334370948908445696}]
I20260928 17:51:47.458323 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.455342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.0.98.3:33039)
0928 17:51:47.455388 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.455390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.455516 (+ 126us) server_negotiation.cc:407] Connection header received
0928 17:51:47.455555 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.455556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.455581 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.455661 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:47.455761 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.456546 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.457367 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.457684 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.457756 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.457939 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.458062 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.458144 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.458223 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.458249 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:51:47.463455 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.461045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49021 (local address 127.0.98.1:41421)
0928 17:51:47.461095 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.461097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.461195 (+ 98us) server_negotiation.cc:407] Connection header received
0928 17:51:47.461255 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.461256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.461268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.461324 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:47.461468 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.461941 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.462718 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.462915 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.463004 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.463200 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.463239 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.463294 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.463362 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.463381 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:51:47.463488 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.460967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.98.3:49021)
0928 17:51:47.461113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.461129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.461191 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.461328 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.461328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.461330 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:47.461390 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.461391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.462030 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.462031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.462632 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.462633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.463006 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.463007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.463071 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.463091 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.463307 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.463309 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.463358 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.463399 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260928 17:51:47.466189 2416 raft_consensus.cc:1261] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.466344 2276 raft_consensus.cc:1261] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.466666 2630 consensus_queue.cc:1035] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:47.466814 2635 consensus_queue.cc:1035] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:47.475991 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.473174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52785 (local address 127.0.98.3:33039)
0928 17:51:47.473222 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:47.473224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:47.473295 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:51:47.473390 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:47.473391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:47.473401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:47.473429 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:47.473607 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.474074 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.475201 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.475493 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.475534 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.475756 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.475785 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.475828 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:47.475909 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:51:47.475927 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:51:47.476048 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:47.473080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33039 (local address 127.0.98.1:52785)
0928 17:51:47.473234 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:47.473245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:47.473321 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:47.473486 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:47.473487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:47.473488 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:47.473536 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.473537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.474152 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.474153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.475101 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:47.475103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:47.475606 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:47.475606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:47.475665 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:47.475679 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:47.475851 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:47.475853 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:51:47.475905 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:51:47.475949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260928 17:51:47.485380 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:47.485514 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:47.486317 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Sending DeleteTablet for 3 replicas of tablet 6a9b5318621d44618cc1d1be236b4ee5
I20260928 17:51:47.486734 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39614
I20260928 17:51:47.486905 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 17:51:47.487008 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50412
W20260928 17:51:47.487048 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:47.487182 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9b5318621d44618cc1d1be236b4ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53058
I20260928 17:51:47.488160 2675 tablet_replica.cc:330] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: stopping tablet replica
I20260928 17:51:47.488317 2676 tablet_replica.cc:330] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: stopping tablet replica
I20260928 17:51:47.488458 2677 tablet_replica.cc:330] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: stopping tablet replica
I20260928 17:51:47.488555 2256 tablet_service.cc:1467] Processing CreateTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 (DEFAULT_TABLE table=test-table [id=0efb13eb7a5e4357892243f062b2dfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:47.488687 2676 raft_consensus.cc:2229] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:47.488740 2676 raft_consensus.cc:2258] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:47.488780 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f186a1113aa1416dbf90e2e63b4ea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.488653 2533 tablet_service.cc:1467] Processing CreateTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 (DEFAULT_TABLE table=test-table [id=0efb13eb7a5e4357892243f062b2dfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:47.488879 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f186a1113aa1416dbf90e2e63b4ea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.490101 2629 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Bootstrap starting.
I20260928 17:51:47.490109 2625 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Bootstrap starting.
I20260928 17:51:47.490494 2629 tablet_bootstrap.cc:654] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.490538 2625 tablet_bootstrap.cc:654] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.490996 2677 raft_consensus.cc:2229] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:47.491103 2677 raft_consensus.cc:2258] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:47.491325 2629 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: No bootstrap required, opened a new log
I20260928 17:51:47.491369 2629 ts_tablet_manager.cc:1397] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:47.491400 2676 ts_tablet_manager.cc:1905] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:47.491499 2629 raft_consensus.cc:348] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.491572 2629 raft_consensus.cc:374] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.491598 2629 raft_consensus.cc:729] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.491640 2629 consensus_queue.cc:260] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.491729 2629 ts_tablet_manager.cc:1428] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:47.490886 2396 tablet_service.cc:1467] Processing CreateTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 (DEFAULT_TABLE table=test-table [id=0efb13eb7a5e4357892243f062b2dfb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:47.492193 2675 raft_consensus.cc:2229] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:47.492242 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f186a1113aa1416dbf90e2e63b4ea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:47.492260 2675 raft_consensus.cc:2258] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:47.492627 2677 ts_tablet_manager.cc:1905] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:47.493635 2626 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Bootstrap starting.
I20260928 17:51:47.494007 2626 tablet_bootstrap.cc:654] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:47.494247 2677 ts_tablet_manager.cc:1918] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:47.494318 2677 log.cc:1196] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/6a9b5318621d44618cc1d1be236b4ee5
I20260928 17:51:47.494590 2677 ts_tablet_manager.cc:1939] T 6a9b5318621d44618cc1d1be236b4ee5 P a155fb2dbd1f481ea24972ad03b35a45: Deleting consensus metadata
I20260928 17:51:47.495009 2105 catalog_manager.cc:4841] TS a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039): tablet 6a9b5318621d44618cc1d1be236b4ee5 (table test-table [id=67e81f3edde54a428a671a778ef3fd17]) successfully deleted
I20260928 17:51:47.495187 2675 ts_tablet_manager.cc:1905] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:47.496737 2675 ts_tablet_manager.cc:1918] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:47.496863 2675 log.cc:1196] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/6a9b5318621d44618cc1d1be236b4ee5
I20260928 17:51:47.497160 2675 ts_tablet_manager.cc:1939] T 6a9b5318621d44618cc1d1be236b4ee5 P 8548eb776c694e55a4bca85fdfae389f: Deleting consensus metadata
I20260928 17:51:47.497504 2676 ts_tablet_manager.cc:1918] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:47.497573 2676 log.cc:1196] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/6a9b5318621d44618cc1d1be236b4ee5
I20260928 17:51:47.497677 2106 catalog_manager.cc:4841] TS 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131): tablet 6a9b5318621d44618cc1d1be236b4ee5 (table test-table [id=67e81f3edde54a428a671a778ef3fd17]) successfully deleted
I20260928 17:51:47.497879 2676 ts_tablet_manager.cc:1939] T 6a9b5318621d44618cc1d1be236b4ee5 P dbd9941751924fcfadd83192ed645b79: Deleting consensus metadata
I20260928 17:51:47.498260 2105 catalog_manager.cc:4841] TS dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421): tablet 6a9b5318621d44618cc1d1be236b4ee5 (table test-table [id=67e81f3edde54a428a671a778ef3fd17]) successfully deleted
I20260928 17:51:47.498525 2625 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: No bootstrap required, opened a new log
I20260928 17:51:47.498591 2625 ts_tablet_manager.cc:1397] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260928 17:51:47.498735 2625 raft_consensus.cc:348] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.498783 2625 raft_consensus.cc:374] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.498796 2625 raft_consensus.cc:729] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.498854 2625 consensus_queue.cc:260] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.498939 2625 ts_tablet_manager.cc:1428] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:47.500444 2626 tablet_bootstrap.cc:492] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: No bootstrap required, opened a new log
I20260928 17:51:47.500488 2626 ts_tablet_manager.cc:1397] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 17:51:47.500646 2626 raft_consensus.cc:348] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.500727 2626 raft_consensus.cc:374] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:47.500751 2626 raft_consensus.cc:729] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Initialized, Role: FOLLOWER
I20260928 17:51:47.500790 2626 consensus_queue.cc:260] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.500882 2626 ts_tablet_manager.cc:1428] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:47.559718 2669 raft_consensus.cc:482] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:47.559844 2669 raft_consensus.cc:504] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.560021 2669 leader_election.cc:288] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:47.560340 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f186a1113aa1416dbf90e2e63b4ea539" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79" is_pre_election: true
I20260928 17:51:47.560427 2276 raft_consensus.cc:2454] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:47.560556 2489 leader_election.cc:302] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:47.560940 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f186a1113aa1416dbf90e2e63b4ea539" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f" is_pre_election: true
I20260928 17:51:47.561044 2416 raft_consensus.cc:2454] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:47.561343 2669 raft_consensus.cc:2781] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:47.561388 2669 raft_consensus.cc:482] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:47.561406 2669 raft_consensus.cc:3037] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.562264 2669 raft_consensus.cc:504] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.562393 2669 leader_election.cc:288] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Requested vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:47.562593 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f186a1113aa1416dbf90e2e63b4ea539" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79"
I20260928 17:51:47.562671 2276 raft_consensus.cc:3037] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.563423 2276 raft_consensus.cc:2454] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:47.563576 2489 leader_election.cc:302] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:47.563902 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f186a1113aa1416dbf90e2e63b4ea539" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f"
I20260928 17:51:47.563974 2416 raft_consensus.cc:3037] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:47.564831 2416 raft_consensus.cc:2454] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:47.565191 2669 raft_consensus.cc:2781] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:47.565248 2669 raft_consensus.cc:686] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Becoming Leader. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Running, Role: LEADER
I20260928 17:51:47.565303 2669 consensus_queue.cc:237] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:47.566001 2117 catalog_manager.cc:5495] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a155fb2dbd1f481ea24972ad03b35a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } health_report { overall_health: HEALTHY } } }
I20260928 17:51:47.589612 2276 raft_consensus.cc:1261] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.589612 2416 raft_consensus.cc:1261] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:47.589852 2669 consensus_queue.cc:1035] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:47.589941 2630 consensus_queue.cc:1035] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 17:51:47.596140 2656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 17:51:49.118243 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:49.118366 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:49.119289 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Sending DeleteTablet for 3 replicas of tablet f186a1113aa1416dbf90e2e63b4ea539
I20260928 17:51:49.119644 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50412
I20260928 17:51:49.119673 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53058
I20260928 17:51:49.119904 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet f186a1113aa1416dbf90e2e63b4ea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39614
I20260928 17:51:49.120129 2706 tablet_replica.cc:330] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: stopping tablet replica
I20260928 17:51:49.120199 2706 raft_consensus.cc:2229] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:49.120239 2706 raft_consensus.cc:2258] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.120337 2707 tablet_replica.cc:330] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: stopping tablet replica
I20260928 17:51:49.120397 2707 raft_consensus.cc:2229] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:49.120433 2707 raft_consensus.cc:2258] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.120711 2707 ts_tablet_manager.cc:1905] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.120720 2706 ts_tablet_manager.cc:1905] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.122371 2707 ts_tablet_manager.cc:1918] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:51:49.122474 2707 log.cc:1196] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/f186a1113aa1416dbf90e2e63b4ea539
I20260928 17:51:49.122710 2707 ts_tablet_manager.cc:1939] T f186a1113aa1416dbf90e2e63b4ea539 P 8548eb776c694e55a4bca85fdfae389f: Deleting consensus metadata
I20260928 17:51:49.123076 2106 catalog_manager.cc:4841] TS 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131): tablet f186a1113aa1416dbf90e2e63b4ea539 (table test-table [id=0efb13eb7a5e4357892243f062b2dfb3]) successfully deleted
I20260928 17:51:49.123163 2706 ts_tablet_manager.cc:1918] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:51:49.123239 2706 log.cc:1196] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/f186a1113aa1416dbf90e2e63b4ea539
I20260928 17:51:49.123466 2706 ts_tablet_manager.cc:1939] T f186a1113aa1416dbf90e2e63b4ea539 P dbd9941751924fcfadd83192ed645b79: Deleting consensus metadata
I20260928 17:51:49.123678 2705 tablet_replica.cc:330] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: stopping tablet replica
I20260928 17:51:49.123749 2705 raft_consensus.cc:2229] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:49.123896 2705 raft_consensus.cc:2258] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.123884 2105 catalog_manager.cc:4841] TS dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421): tablet f186a1113aa1416dbf90e2e63b4ea539 (table test-table [id=0efb13eb7a5e4357892243f062b2dfb3]) successfully deleted
I20260928 17:51:49.124228 2705 ts_tablet_manager.cc:1905] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.125399 2705 ts_tablet_manager.cc:1918] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:51:49.125452 2705 log.cc:1196] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/f186a1113aa1416dbf90e2e63b4ea539
I20260928 17:51:49.125628 2705 ts_tablet_manager.cc:1939] T f186a1113aa1416dbf90e2e63b4ea539 P a155fb2dbd1f481ea24972ad03b35a45: Deleting consensus metadata
I20260928 17:51:49.125967 2105 catalog_manager.cc:4841] TS a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039): tablet f186a1113aa1416dbf90e2e63b4ea539 (table test-table [id=0efb13eb7a5e4357892243f062b2dfb3]) successfully deleted
Sep 28 17:51:49 dist-test-slave-zqlt krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:51:49 dist-test-slave-zqlt krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:49.151721 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.139794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.98.62:37487)
0928 17:51:49.139959 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.139961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.139998 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:51:49.140066 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.140067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.140079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.140225 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:49.140766 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.141292 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.142029 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.142222 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.142281 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.149052 (+ 6771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:49.149053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:49.149054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:49.149093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:49.150593 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:49.150933 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:49.150935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:49.150935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:49.151005 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:49.151186 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:49.151186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:49.151187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:49.151305 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:49.151416 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.151507 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.151581 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260928 17:51:49.153451 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:49.153617 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:49.155073 2396 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 (DEFAULT_TABLE table=test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.155057 2256 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 (DEFAULT_TABLE table=test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.155153 2533 tablet_service.cc:1467] Processing CreateTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 (DEFAULT_TABLE table=test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.155211 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f7ac68d4541c4ab72fcd397e25d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.155215 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f7ac68d4541c4ab72fcd397e25d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.155282 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7f7ac68d4541c4ab72fcd397e25d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.156374 2716 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Bootstrap starting.
I20260928 17:51:49.156730 2717 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Bootstrap starting.
I20260928 17:51:49.156855 2716 tablet_bootstrap.cc:654] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.157105 2717 tablet_bootstrap.cc:654] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.157397 2718 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Bootstrap starting.
I20260928 17:51:49.157585 2716 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: No bootstrap required, opened a new log
I20260928 17:51:49.157655 2717 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: No bootstrap required, opened a new log
I20260928 17:51:49.157680 2716 ts_tablet_manager.cc:1397] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:49.157704 2717 ts_tablet_manager.cc:1397] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:49.157747 2718 tablet_bootstrap.cc:654] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.157815 2716 raft_consensus.cc:348] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.157853 2717 raft_consensus.cc:348] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.157905 2716 raft_consensus.cc:374] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.157920 2717 raft_consensus.cc:374] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.157932 2716 raft_consensus.cc:729] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.157948 2717 raft_consensus.cc:729] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.157997 2717 consensus_queue.cc:260] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.157991 2716 consensus_queue.cc:260] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.158105 2716 ts_tablet_manager.cc:1428] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:49.158161 2717 ts_tablet_manager.cc:1428] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:49.158746 2718 tablet_bootstrap.cc:492] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: No bootstrap required, opened a new log
I20260928 17:51:49.158800 2718 ts_tablet_manager.cc:1397] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:49.158923 2718 raft_consensus.cc:348] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.158998 2718 raft_consensus.cc:374] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.159022 2718 raft_consensus.cc:729] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.159072 2718 consensus_queue.cc:260] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.159171 2718 ts_tablet_manager.cc:1428] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:49.290684 2722 raft_consensus.cc:482] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:49.290789 2722 raft_consensus.cc:504] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.291152 2722 leader_election.cc:288] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039)
I20260928 17:51:49.294900 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.291376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.98.2:40131)
0928 17:51:49.291561 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:49.291573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:49.291632 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:49.291861 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:49.291862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:49.291865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:49.291944 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.291946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.292825 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.292826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.294036 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.294039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.294644 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.294717 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.294723 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:51:49.294779 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:51:49.294825 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":132,"threads_started":1}
I20260928 17:51:49.294919 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.291445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40131 (local address 127.0.98.1:41421)
0928 17:51:49.291628 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.291630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.291657 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:51:49.291693 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.291694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.291705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.291836 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:49.292058 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.292745 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294145 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.294559 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294618 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.294628 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.294796 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.294842 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":124,"threads_started":1}
I20260928 17:51:49.295112 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.291654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33039 (local address 127.0.98.2:56849)
0928 17:51:49.291762 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:49.291770 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:49.291811 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:49.292013 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:49.292014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:49.292016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:49.292349 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.292351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.293110 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.293111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.294176 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.294177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.294828 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.294917 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.294921 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:49.294978 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:51:49.295023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":69,"threads_started":1}
I20260928 17:51:49.295197 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f7ac68d4541c4ab72fcd397e25d20" candidate_uuid: "8548eb776c694e55a4bca85fdfae389f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79" is_pre_election: true
I20260928 17:51:49.295287 2276 raft_consensus.cc:2454] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8548eb776c694e55a4bca85fdfae389f in term 0.
I20260928 17:51:49.295305 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.291739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56849 (local address 127.0.98.3:33039)
0928 17:51:49.291885 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.291887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.291896 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:51:49.291924 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.291925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.291933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.291987 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:49.292426 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.293024 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294253 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.294683 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.294759 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.294772 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.295084 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.295137 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 17:51:49.295459 2349 leader_election.cc:302] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8548eb776c694e55a4bca85fdfae389f, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:49.295503 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f7ac68d4541c4ab72fcd397e25d20" candidate_uuid: "8548eb776c694e55a4bca85fdfae389f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a155fb2dbd1f481ea24972ad03b35a45" is_pre_election: true
I20260928 17:51:49.295591 2556 raft_consensus.cc:2454] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8548eb776c694e55a4bca85fdfae389f in term 0.
I20260928 17:51:49.295624 2722 raft_consensus.cc:2781] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:49.295693 2722 raft_consensus.cc:482] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:49.295727 2722 raft_consensus.cc:3037] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.296492 2722 raft_consensus.cc:504] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.296645 2722 leader_election.cc:288] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [CANDIDATE]: Term 1 election: Requested vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039)
I20260928 17:51:49.296849 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f7ac68d4541c4ab72fcd397e25d20" candidate_uuid: "8548eb776c694e55a4bca85fdfae389f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79"
I20260928 17:51:49.296854 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7f7ac68d4541c4ab72fcd397e25d20" candidate_uuid: "8548eb776c694e55a4bca85fdfae389f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a155fb2dbd1f481ea24972ad03b35a45"
I20260928 17:51:49.296921 2276 raft_consensus.cc:3037] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.296939 2556 raft_consensus.cc:3037] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.297504 2276 raft_consensus.cc:2454] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8548eb776c694e55a4bca85fdfae389f in term 1.
I20260928 17:51:49.297607 2556 raft_consensus.cc:2454] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8548eb776c694e55a4bca85fdfae389f in term 1.
I20260928 17:51:49.297686 2349 leader_election.cc:302] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8548eb776c694e55a4bca85fdfae389f, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:49.297771 2722 raft_consensus.cc:2781] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:49.297905 2722 raft_consensus.cc:686] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 1 LEADER]: Becoming Leader. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Running, Role: LEADER
I20260928 17:51:49.297986 2722 consensus_queue.cc:237] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.298703 2117 catalog_manager.cc:5495] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8548eb776c694e55a4bca85fdfae389f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8548eb776c694e55a4bca85fdfae389f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } health_report { overall_health: UNKNOWN } } }
I20260928 17:51:49.307247 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.304542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.0.98.2:36131)
0928 17:51:49.304713 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.304715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.304722 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:51:49.304763 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.304764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.304781 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.304869 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:49.305024 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.305562 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.306329 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.306551 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.306641 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.306859 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.306968 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.307059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.307100 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.307120 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260928 17:51:49.311947 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.309561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60029 (local address 127.0.98.1:41421)
0928 17:51:49.309638 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.309640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.309729 (+ 89us) server_negotiation.cc:407] Connection header received
0928 17:51:49.309796 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.309797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.309809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.309879 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:49.310017 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.310471 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.311235 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.311434 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.311512 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.311691 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.311727 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.311787 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.311848 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.311871 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 17:51:49.311970 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.309465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.98.2:60029)
0928 17:51:49.309654 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:49.309665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:49.309744 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:49.309880 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:49.309881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:49.309883 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:49.309942 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.309943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.310552 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.310554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.311145 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.311146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.311509 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.311510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.311575 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.311596 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.311788 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.311790 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:51:49.311843 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:51:49.311888 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260928 17:51:49.314193 2276 raft_consensus.cc:1261] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Refusing update from remote peer 8548eb776c694e55a4bca85fdfae389f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:49.314193 2556 raft_consensus.cc:1261] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Refusing update from remote peer 8548eb776c694e55a4bca85fdfae389f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:49.314476 2727 consensus_queue.cc:1035] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:49.314669 2722 consensus_queue.cc:1035] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:49.321887 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.319399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60643 (local address 127.0.98.2:36131)
0928 17:51:49.319445 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.319447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.319467 (+ 20us) server_negotiation.cc:407] Connection header received
0928 17:51:49.319541 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.319542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.319553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.319609 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:49.319721 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.320286 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.321060 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.321390 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.321460 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.321627 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.321662 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.321718 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.321798 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.321821 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:51:49.321921 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.319268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36131 (local address 127.0.98.1:60643)
0928 17:51:49.319411 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:49.319425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:49.319480 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:49.319610 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:49.319611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:49.319613 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:51:49.319667 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.319669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.320364 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.320365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.320977 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:49.320979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.321470 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.321472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:49.321540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.321547 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.321729 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.321731 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:51:49.321793 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:51:49.321852 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 17:51:49.328924 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:49.329059 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:49.329834 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Sending DeleteTablet for 3 replicas of tablet ac7f7ac68d4541c4ab72fcd397e25d20
I20260928 17:51:49.330153 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39614
I20260928 17:51:49.330202 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50412
I20260928 17:51:49.330216 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7f7ac68d4541c4ab72fcd397e25d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53058
I20260928 17:51:49.330255 2707 tablet_replica.cc:330] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: stopping tablet replica
I20260928 17:51:49.330313 2707 raft_consensus.cc:2229] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:49.330327 2705 tablet_replica.cc:330] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: stopping tablet replica
I20260928 17:51:49.330379 2705 raft_consensus.cc:2229] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:49.330415 2705 raft_consensus.cc:2258] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.330428 2707 raft_consensus.cc:2258] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.330523 2706 tablet_replica.cc:330] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: stopping tablet replica
I20260928 17:51:49.330580 2706 raft_consensus.cc:2229] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:49.330616 2706 raft_consensus.cc:2258] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:49.330681 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:49.330835 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:49.331012 2706 ts_tablet_manager.cc:1905] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.331451 2707 ts_tablet_manager.cc:1905] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.332330 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 976a12375bab43fb8b490345abf28cf0 (DEFAULT_TABLE table=test-table [id=00e7cee66be54d74abb94ad20f206d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.332444 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a12375bab43fb8b490345abf28cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.332795 2706 ts_tablet_manager.cc:1918] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:49.332849 2706 log.cc:1196] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/ac7f7ac68d4541c4ab72fcd397e25d20
I20260928 17:51:49.333038 2707 ts_tablet_manager.cc:1918] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:49.333087 2707 log.cc:1196] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/ac7f7ac68d4541c4ab72fcd397e25d20
I20260928 17:51:49.333101 2706 ts_tablet_manager.cc:1939] T ac7f7ac68d4541c4ab72fcd397e25d20 P dbd9941751924fcfadd83192ed645b79: Deleting consensus metadata
I20260928 17:51:49.333314 2707 ts_tablet_manager.cc:1939] T ac7f7ac68d4541c4ab72fcd397e25d20 P 8548eb776c694e55a4bca85fdfae389f: Deleting consensus metadata
I20260928 17:51:49.333575 2105 catalog_manager.cc:4841] TS dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421): tablet ac7f7ac68d4541c4ab72fcd397e25d20 (table test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]) successfully deleted
I20260928 17:51:49.333658 2705 ts_tablet_manager.cc:1905] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:49.333817 2716 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Bootstrap starting.
I20260928 17:51:49.334125 2716 tablet_bootstrap.cc:654] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.334455 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 976a12375bab43fb8b490345abf28cf0 (DEFAULT_TABLE table=test-table [id=00e7cee66be54d74abb94ad20f206d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.334547 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a12375bab43fb8b490345abf28cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.335098 2705 ts_tablet_manager.cc:1918] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:51:49.335148 2705 log.cc:1196] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/ac7f7ac68d4541c4ab72fcd397e25d20
I20260928 17:51:49.335353 2705 ts_tablet_manager.cc:1939] T ac7f7ac68d4541c4ab72fcd397e25d20 P a155fb2dbd1f481ea24972ad03b35a45: Deleting consensus metadata
I20260928 17:51:49.335897 2718 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79: Bootstrap starting.
I20260928 17:51:49.336079 2105 catalog_manager.cc:4841] TS a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039): tablet ac7f7ac68d4541c4ab72fcd397e25d20 (table test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]) successfully deleted
I20260928 17:51:49.336238 2718 tablet_bootstrap.cc:654] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.336483 2716 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: No bootstrap required, opened a new log
I20260928 17:51:49.336536 2716 ts_tablet_manager.cc:1397] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:51:49.336649 2716 raft_consensus.cc:348] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.336740 2716 raft_consensus.cc:374] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.336771 2716 raft_consensus.cc:729] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.336807 2716 consensus_queue.cc:260] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.336908 2716 ts_tablet_manager.cc:1428] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:49.336938 2718 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79: No bootstrap required, opened a new log
I20260928 17:51:49.336987 2718 ts_tablet_manager.cc:1397] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:49.337126 2718 raft_consensus.cc:348] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.337208 2718 raft_consensus.cc:374] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.337230 2718 raft_consensus.cc:729] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9941751924fcfadd83192ed645b79, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.337275 2718 consensus_queue.cc:260] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.337365 2718 ts_tablet_manager.cc:1428] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:49.337697 2106 catalog_manager.cc:4841] TS 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131): tablet ac7f7ac68d4541c4ab72fcd397e25d20 (table test-table [id=42f19fb7dfb64b6ea3d86b597f57afb6]) successfully deleted
I20260928 17:51:49.337714 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 976a12375bab43fb8b490345abf28cf0 (DEFAULT_TABLE table=test-table [id=00e7cee66be54d74abb94ad20f206d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:49.337810 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a12375bab43fb8b490345abf28cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:49.338855 2717 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Bootstrap starting.
I20260928 17:51:49.339327 2717 tablet_bootstrap.cc:654] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:49.340016 2717 tablet_bootstrap.cc:492] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: No bootstrap required, opened a new log
I20260928 17:51:49.340078 2717 ts_tablet_manager.cc:1397] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:49.340276 2717 raft_consensus.cc:348] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.340354 2717 raft_consensus.cc:374] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:49.340378 2717 raft_consensus.cc:729] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8548eb776c694e55a4bca85fdfae389f, State: Initialized, Role: FOLLOWER
I20260928 17:51:49.340426 2717 consensus_queue.cc:260] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.340543 2717 ts_tablet_manager.cc:1428] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:49.442375 2696 raft_consensus.cc:482] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:49.442474 2696 raft_consensus.cc:504] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.442653 2696 leader_election.cc:288] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:49.442925 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a12375bab43fb8b490345abf28cf0" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f" is_pre_election: true
I20260928 17:51:49.442915 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a12375bab43fb8b490345abf28cf0" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79" is_pre_election: true
I20260928 17:51:49.443017 2276 raft_consensus.cc:2454] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:49.443017 2415 raft_consensus.cc:2454] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 0.
I20260928 17:51:49.443197 2489 leader_election.cc:302] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:49.443356 2696 raft_consensus.cc:2781] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:49.443444 2696 raft_consensus.cc:482] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:49.443470 2696 raft_consensus.cc:3037] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.444525 2696 raft_consensus.cc:504] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.444695 2696 leader_election.cc:288] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Requested vote from peers dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421), 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131)
I20260928 17:51:49.444885 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a12375bab43fb8b490345abf28cf0" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd9941751924fcfadd83192ed645b79"
I20260928 17:51:49.444973 2276 raft_consensus.cc:3037] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.444923 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a12375bab43fb8b490345abf28cf0" candidate_uuid: "a155fb2dbd1f481ea24972ad03b35a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8548eb776c694e55a4bca85fdfae389f"
I20260928 17:51:49.445027 2415 raft_consensus.cc:3037] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:49.445683 2415 raft_consensus.cc:2454] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:49.445737 2276 raft_consensus.cc:2454] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a155fb2dbd1f481ea24972ad03b35a45 in term 1.
I20260928 17:51:49.445922 2489 leader_election.cc:302] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a155fb2dbd1f481ea24972ad03b35a45, dbd9941751924fcfadd83192ed645b79; no voters:
I20260928 17:51:49.446028 2696 raft_consensus.cc:2781] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:49.446113 2696 raft_consensus.cc:686] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Becoming Leader. State: Replica: a155fb2dbd1f481ea24972ad03b35a45, State: Running, Role: LEADER
I20260928 17:51:49.446171 2696 consensus_queue.cc:237] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } }
I20260928 17:51:49.446800 2119 catalog_manager.cc:5495] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a155fb2dbd1f481ea24972ad03b35a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a155fb2dbd1f481ea24972ad03b35a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33039 } health_report { overall_health: HEALTHY } } }
I20260928 17:51:49.460043 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:49.457829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.3:33039)
0928 17:51:49.457868 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:49.457870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:49.457878 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:51:49.457922 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:49.457923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:49.457935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:49.457989 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:49.458113 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.458595 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.459316 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:49.459528 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:49.459622 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:49.459776 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:49.459815 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:49.459873 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:49.459935 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 17:51:49.459957 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:51:49.462514 2276 raft_consensus.cc:1261] T 976a12375bab43fb8b490345abf28cf0 P dbd9941751924fcfadd83192ed645b79 [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:49.462743 2415 raft_consensus.cc:1261] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Refusing update from remote peer a155fb2dbd1f481ea24972ad03b35a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:49.462776 2700 consensus_queue.cc:1035] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd9941751924fcfadd83192ed645b79" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:49.462955 2700 consensus_queue.cc:1035] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8548eb776c694e55a4bca85fdfae389f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 17:51:49.467217 2687 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 17:51:50.983639 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:50.983757 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:50.984663 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213da64453dd44778342760ec1eef153: Sending DeleteTablet for 3 replicas of tablet 976a12375bab43fb8b490345abf28cf0
I20260928 17:51:50.984982 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a12375bab43fb8b490345abf28cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53058
I20260928 17:51:50.984982 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a12375bab43fb8b490345abf28cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39614
I20260928 17:51:50.985085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2191
I20260928 17:51:50.985162 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a12375bab43fb8b490345abf28cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50412
I20260928 17:51:50.987066 2768 tablet_replica.cc:330] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: stopping tablet replica
I20260928 17:51:50.987142 2768 raft_consensus.cc:2229] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:50.987212 2768 raft_consensus.cc:2258] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:50.987329 2769 tablet_replica.cc:330] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: stopping tablet replica
I20260928 17:51:50.987442 2769 raft_consensus.cc:2229] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:50.987466 2768 ts_tablet_manager.cc:1905] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:50.987526 2769 raft_consensus.cc:2258] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:50.987794 2769 ts_tablet_manager.cc:1905] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:50.989059 2768 ts_tablet_manager.cc:1918] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:51:50.989137 2768 log.cc:1196] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/976a12375bab43fb8b490345abf28cf0
I20260928 17:51:50.989398 2768 ts_tablet_manager.cc:1939] T 976a12375bab43fb8b490345abf28cf0 P a155fb2dbd1f481ea24972ad03b35a45: Deleting consensus metadata
I20260928 17:51:50.989739 2105 catalog_manager.cc:4841] TS a155fb2dbd1f481ea24972ad03b35a45 (127.0.98.3:33039): tablet 976a12375bab43fb8b490345abf28cf0 (table test-table [id=00e7cee66be54d74abb94ad20f206d0e]) successfully deleted
I20260928 17:51:50.989814 2769 ts_tablet_manager.cc:1918] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 17:51:50.989880 2769 log.cc:1196] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/976a12375bab43fb8b490345abf28cf0
I20260928 17:51:50.990093 2769 ts_tablet_manager.cc:1939] T 976a12375bab43fb8b490345abf28cf0 P 8548eb776c694e55a4bca85fdfae389f: Deleting consensus metadata
I20260928 17:51:50.990396 2106 catalog_manager.cc:4841] TS 8548eb776c694e55a4bca85fdfae389f (127.0.98.2:36131): tablet 976a12375bab43fb8b490345abf28cf0 (table test-table [id=00e7cee66be54d74abb94ad20f206d0e]) successfully deleted
W20260928 17:51:50.992379 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41421 (error 108)
I20260928 17:51:50.992492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2331
I20260928 17:51:50.993451 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:50.993265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41421 (local address 127.0.0.1:50426)
0928 17:51:50.993381 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:50.993418 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
W20260928 17:51:50.993597 2105 catalog_manager.cc:4584] TS dbd9941751924fcfadd83192ed645b79 (127.0.98.1:41421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 976a12375bab43fb8b490345abf28cf0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41421: connect: Connection refused (error 111)
I20260928 17:51:50.999012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2471
I20260928 17:51:51.004455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2087
2026-09-28T17:51:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5961 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:51 dist-test-slave-zqlt krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:51 dist-test-slave-zqlt krb5kdc[2774](info): set up 2 sockets
Sep 28 17:51:51 dist-test-slave-zqlt krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260928 17:51:52.047899 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:52.163000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:52.246640 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:52.246836 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:52.246865 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:52.247954 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:52.247987 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:52.248001 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:52.248014 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:52.248042 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:52.249586 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:52.249776 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:52.249945 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:52.254726 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.254724 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.254724 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:52.255007 2787 server_base.cc:1034] running on GCE node
I20260928 17:51:52.255288 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:52.255621 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:52.256829 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790617912256805 us; error 41 us; skew 500 ppm
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:52.259470 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:52.259856 2787 webserver.cc:460] Webserver started at http://127.0.98.62:43379/ using document root <none> and password file <none>
I20260928 17:51:52.260053 2787 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:52.260149 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:52.260268 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:52.261322 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "77a115ce858d4efc83fc7b43fc6ce35b"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "6806a26a9a4da16ad1f1395b23b73651"
server_key_iv: "9db9129ba4f8621632e55dbec0a9f26d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.261620 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "77a115ce858d4efc83fc7b43fc6ce35b"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "6806a26a9a4da16ad1f1395b23b73651"
server_key_iv: "9db9129ba4f8621632e55dbec0a9f26d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.262928 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:51:52.263628 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.263931 2787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:52.264051 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "77a115ce858d4efc83fc7b43fc6ce35b"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "6806a26a9a4da16ad1f1395b23b73651"
server_key_iv: "9db9129ba4f8621632e55dbec0a9f26d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.264176 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:52.305445 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:52.307366 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:52.307533 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:52.311672 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36489
I20260928 17:51:52.311967 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36489 every 8 connection(s)
I20260928 17:51:52.312069 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:52.312822 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:52.315111 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: Bootstrap starting.
I20260928 17:51:52.315713 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:52.316018 2855 log.cc:824] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:52.316748 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: No bootstrap required, opened a new log
I20260928 17:51:52.318168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2787
I20260928 17:51:52.318265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:52.318435 392 external_mini_cluster.cc:1428] Setting key 422c8840b0678b40fbdb1371099d1c7b
I20260928 17:51:52.318607 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260928 17:51:52.318786 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:52.318822 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a115ce858d4efc83fc7b43fc6ce35b, State: Initialized, Role: FOLLOWER
I20260928 17:51:52.318994 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260928 17:51:52.319097 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:52.319134 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:52.319195 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:52.319757 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260928 17:51:52.319878 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77a115ce858d4efc83fc7b43fc6ce35b; no voters:
I20260928 17:51:52.320082 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:52.320142 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:52.320298 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [term 1 LEADER]: Becoming Leader. State: Replica: 77a115ce858d4efc83fc7b43fc6ce35b, State: Running, Role: LEADER
I20260928 17:51:52.320369 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:52.320461 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260928 17:51:52.321091 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260928 17:51:52.321101 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77a115ce858d4efc83fc7b43fc6ce35b. Latest consensus state: current_term: 1 leader_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77a115ce858d4efc83fc7b43fc6ce35b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260928 17:51:52.321283 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:52.321296 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b [sys.catalog]: This master's current role is: LEADER
W20260928 17:51:52.322310 2875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:51:52.322373 2875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:51:52.322532 2869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:52.322700 2869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:52.324286 2869 catalog_manager.cc:1347] Generated new cluster ID: c96ebfdb6abf4ce8aa0287807634f2c1
I20260928 17:51:52.324333 2869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:52.330699 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.319232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.0.98.62:36489)
0928 17:51:52.319411 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:52.319419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:52.319446 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:51:52.319567 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:52.319574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:52.319617 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:52.319761 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:52.320574 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.322232 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.323174 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.323807 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.323917 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.326685 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:52.326688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:52.326689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:52.326701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:52.328770 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.329169 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.329173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.329173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.329284 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.329457 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.329457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.329458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.329760 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:52.329908 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:52.330317 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 17:51:52.330459 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260928 17:51:52.344312 2869 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:52.344925 2869 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:52.348784 2869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77a115ce858d4efc83fc7b43fc6ce35b: Generated new TSK 0
I20260928 17:51:52.348994 2869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:52.379273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:52.465493 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:52.465716 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:52.465746 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:52.467028 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:52.467072 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:52.467098 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:52.468811 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:52.469065 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:52.469296 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:52.474226 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.474226 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.474260 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:52.474534 2881 server_base.cc:1034] running on GCE node
I20260928 17:51:52.474742 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:52.474932 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:52.476146 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790617912476128 us; error 37 us; skew 500 ppm
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:52.478627 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:52.479100 2881 webserver.cc:460] Webserver started at http://127.0.98.1:35935/ using document root <none> and password file <none>
I20260928 17:51:52.479333 2881 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:52.479401 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:52.479557 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:52.480690 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "248b9c99c67c4f4ca8caa7cda7f22f43"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "dd25eb8c0caa1e65a726945821437714"
server_key_iv: "c1867e136dbb846bf828518d8680fbac"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.481061 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "248b9c99c67c4f4ca8caa7cda7f22f43"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "dd25eb8c0caa1e65a726945821437714"
server_key_iv: "c1867e136dbb846bf828518d8680fbac"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.482383 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:52.483094 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.483260 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:52.483351 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "248b9c99c67c4f4ca8caa7cda7f22f43"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "dd25eb8c0caa1e65a726945821437714"
server_key_iv: "c1867e136dbb846bf828518d8680fbac"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.483433 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:52.506173 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:52.508234 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:52.508401 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:52.508649 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:52.508960 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:52.509024 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.509065 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:52.509114 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.514796 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40457
I20260928 17:51:52.514837 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40457 every 8 connection(s)
I20260928 17:51:52.515153 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:51:52.515707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 2881
I20260928 17:51:52.515810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:52.516008 392 external_mini_cluster.cc:1428] Setting key f70fc1a62680344f8d0cbe720b695d3e
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:51:52.526716 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.515898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34421 (local address 127.0.98.62:36489)
0928 17:51:52.515960 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:52.515962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:52.516229 (+ 267us) server_negotiation.cc:407] Connection header received
0928 17:51:52.517609 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:52.517610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:52.517621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:52.517664 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:52.518522 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.519319 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.520943 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.521147 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.521205 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.523596 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:52.523598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:52.523598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:52.523606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:52.525149 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.525565 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.525566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.525566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.525627 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.526014 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.526016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.526016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.526136 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:52.526215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:52.526504 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 17:51:52.526624 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:51:52.526854 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.515875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.1:34421)
0928 17:51:52.516076 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:52.516148 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:52.517493 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:52.517755 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:52.517760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:52.518165 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:52.518414 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.518419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.519411 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.519413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.520849 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.520851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.521235 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.521237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.521392 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.521908 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:52.521919 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:52.523488 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:52.525246 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.525253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.525255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.525480 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.525835 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.525835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.525835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.525936 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.526206 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:52.526212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:52.526347 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 17:51:52.526459 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 17:51:52.526630 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 17:51:52.527405 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260928 17:51:52.527513 3010 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:52.527693 3010 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260928 17:51:52.528307 2819 ts_manager.cc:194] Registered new tserver with Master: 248b9c99c67c4f4ca8caa7cda7f22f43 (127.0.98.1:40457)
I20260928 17:51:52.528932 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:52.543532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:52.623780 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:52.624013 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:52.624037 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:52.625345 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:52.625391 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:52.625419 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:52.627097 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:52.627354 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:52.627650 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:52.632748 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.632750 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.632776 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:52.632975 3015 server_base.cc:1034] running on GCE node
I20260928 17:51:52.633244 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:52.633500 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:52.634662 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790617912634656 us; error 36 us; skew 500 ppm
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:52.637312 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:52.637760 3015 webserver.cc:460] Webserver started at http://127.0.98.2:34449/ using document root <none> and password file <none>
I20260928 17:51:52.637998 3015 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:52.638068 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:52.638222 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:52.639281 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "43f55d56bfc94726afa547dbf6e29613"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "fabfa9082d87b98afa3c51f75a38c17e"
server_key_iv: "7159aab50c36d1ac2b08e1cf9c48056d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.639653 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43f55d56bfc94726afa547dbf6e29613"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "fabfa9082d87b98afa3c51f75a38c17e"
server_key_iv: "7159aab50c36d1ac2b08e1cf9c48056d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.641065 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:52.641790 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.641953 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:52.642060 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "43f55d56bfc94726afa547dbf6e29613"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "fabfa9082d87b98afa3c51f75a38c17e"
server_key_iv: "7159aab50c36d1ac2b08e1cf9c48056d"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.642135 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:52.657527 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:52.659858 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:52.660034 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:52.660300 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:52.660614 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:52.660667 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.660709 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:52.660738 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.666290 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34983
I20260928 17:51:52.666363 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34983 every 8 connection(s)
I20260928 17:51:52.666683 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:52.668233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3015
I20260928 17:51:52.668329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:52.668502 392 external_mini_cluster.cc:1428] Setting key d095832207ad93a0d0167bdd7012eb54
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:51:52.677249 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.667533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60937 (local address 127.0.98.62:36489)
0928 17:51:52.667613 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:52.667616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:52.667787 (+ 171us) server_negotiation.cc:407] Connection header received
0928 17:51:52.668928 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:52.668929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:52.668937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:52.669001 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:52.669673 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.670429 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.671878 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.672246 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.672321 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.674455 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:52.674456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:52.674456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:52.674463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:52.675793 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.676266 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.676267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.676268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.676324 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.676575 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.676576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.676576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.676687 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:52.676769 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:52.677120 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 17:51:52.677186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:51:52.677412 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.667511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.2:60937)
0928 17:51:52.667663 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:52.667704 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:52.668839 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:52.669086 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:52.669090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:52.669389 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:52.669580 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.669585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.670557 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.670558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.671793 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.671795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.672323 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.672323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.672423 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.672932 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:52.672941 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:52.674349 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:52.675888 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.675895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.675897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.676175 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.676390 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.676391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.676391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.676506 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.676788 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:52.676796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:52.676965 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 17:51:52.677094 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 17:51:52.677213 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 17:51:52.677893 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260928 17:51:52.677999 3144 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:52.678166 3144 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260928 17:51:52.678545 2819 ts_manager.cc:194] Registered new tserver with Master: 43f55d56bfc94726afa547dbf6e29613 (127.0.98.2:34983)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 17:51:52.678902 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:52.693212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:52.775136 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:52.775357 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:52.775380 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:52.776657 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:52.776703 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:52.776732 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:52.778424 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:52.778710 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:52.778962 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:52.784021 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:52.784126 3149 server_base.cc:1034] running on GCE node
W20260928 17:51:52.783946 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:52.783946 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:52.784436 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:52.784688 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:52.785848 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790617912785842 us; error 38 us; skew 500 ppm
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:52.788503 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:52.788923 3149 webserver.cc:460] Webserver started at http://127.0.98.3:35085/ using document root <none> and password file <none>
I20260928 17:51:52.789162 3149 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:52.789230 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:52.789366 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:52.790380 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "56c18acdf4fd4b93801833844f112dc6"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "d8bfb3887795d1ffb3aa15bfa5e2f772"
server_key_iv: "cf6db06d91a823466537fe9c1720d718"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.790745 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56c18acdf4fd4b93801833844f112dc6"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "d8bfb3887795d1ffb3aa15bfa5e2f772"
server_key_iv: "cf6db06d91a823466537fe9c1720d718"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.791998 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:51:52.792716 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.792874 3149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:52.792968 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "56c18acdf4fd4b93801833844f112dc6"
format_stamp: "Formatted at 2026-09-28 17:51:52 on dist-test-slave-zqlt"
server_key: "d8bfb3887795d1ffb3aa15bfa5e2f772"
server_key_iv: "cf6db06d91a823466537fe9c1720d718"
server_key_version: "encryptionkey@0"
I20260928 17:51:52.793053 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:52.814273 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:52.816756 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:52.816929 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:52.817179 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:52.817494 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:52.817548 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.817592 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:52.817620 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:52.823900 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39981
I20260928 17:51:52.823938 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39981 every 8 connection(s)
I20260928 17:51:52.824333 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:52.828192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3149
I20260928 17:51:52.828300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:52.828509 392 external_mini_cluster.cc:1428] Setting key f29599a25dbffbd599803f958fc8dd58
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:52.834447 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.825169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34867 (local address 127.0.98.62:36489)
0928 17:51:52.825234 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:52.825237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:52.825403 (+ 166us) server_negotiation.cc:407] Connection header received
0928 17:51:52.826394 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:52.826396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:52.826408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:52.826478 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:52.827186 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.827691 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.829334 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.829542 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.829631 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.831777 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:52.831779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:52.831779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:52.831787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:52.833002 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.833443 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.833445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.833445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.833506 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:52.833773 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:52.833774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:52.833775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:52.833898 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:52.833973 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:52.834266 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 17:51:52.834352 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:51:52.834674 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:52.825052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.98.3:34867)
0928 17:51:52.825259 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:52.825307 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:52.826270 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:52.826553 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:52.826557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:52.826874 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:52.827084 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.827088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.827798 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.827799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.829233 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:52.829236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:52.829640 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:52.829641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:52.829784 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:52.830279 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:52.830303 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:52.831673 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:52.833090 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.833097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.833100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.833348 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.833584 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:52.833585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:52.833585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:52.833706 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:52.833980 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:52.833987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:52.834132 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 17:51:52.834244 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 17:51:52.834411 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260928 17:51:52.835199 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36489
I20260928 17:51:52.835310 3278 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:52.835506 3278 heartbeater.cc:511] Master 127.0.98.62:36489 requested a full tablet report, sending...
I20260928 17:51:52.835948 2819 ts_manager.cc:194] Registered new tserver with Master: 56c18acdf4fd4b93801833844f112dc6 (127.0.98.3:39981)
I20260928 17:51:52.836395 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34867
I20260928 17:51:52.840288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:51:52.851841 2859 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:51:52.850871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.0.98.62:36489)
0928 17:51:52.850939 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:52.850941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:52.851080 (+ 139us) server_negotiation.cc:407] Connection header received
0928 17:51:52.851142 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:52.851144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:52.851153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:52.851195 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:52.851765 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53538: BlockingRecv error: recv got EOF from 127.0.0.1:53538 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:51:52.852671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2881
I20260928 17:51:52.856827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3015
I20260928 17:51:52.861430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3149
I20260928 17:51:52.866665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 2787
2026-09-28T17:51:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1861 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[3290](info): set up 2 sockets
Sep 28 17:51:52 dist-test-slave-zqlt krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260928 17:51:53.911865 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:53 dist-test-slave-zqlt krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:54.030380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:54.111107 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:54.111317 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:54.111338 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:54.112552 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:54.112596 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:54.112610 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:54.112622 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:54.112634 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:54.114159 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:54.114426 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:54.114666 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:54.119611 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:54.119644 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:54.119616 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:54.119994 3303 server_base.cc:1034] running on GCE node
I20260928 17:51:54.120265 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:54.120573 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:54.121716 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790617914121709 us; error 30 us; skew 500 ppm
Sep 28 17:51:54 dist-test-slave-zqlt krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:54.124225 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:54.124714 3303 webserver.cc:460] Webserver started at http://127.0.98.62:36073/ using document root <none> and password file <none>
I20260928 17:51:54.124959 3303 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:54.125026 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:54.125175 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:54.126240 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "a26cc65660494007b06f22ec678655ce"
format_stamp: "Formatted at 2026-09-28 17:51:54 on dist-test-slave-zqlt"
server_key: "7132c43bc914c1ed0c4d5f037163a986"
server_key_iv: "01f51796e9b7ca273140ff3e7de32277"
server_key_version: "encryptionkey@0"
I20260928 17:51:54.126606 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "a26cc65660494007b06f22ec678655ce"
format_stamp: "Formatted at 2026-09-28 17:51:54 on dist-test-slave-zqlt"
server_key: "7132c43bc914c1ed0c4d5f037163a986"
server_key_iv: "01f51796e9b7ca273140ff3e7de32277"
server_key_version: "encryptionkey@0"
I20260928 17:51:54.127943 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:51:54.128670 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:54.128842 3303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:54.128942 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "a26cc65660494007b06f22ec678655ce"
format_stamp: "Formatted at 2026-09-28 17:51:54 on dist-test-slave-zqlt"
server_key: "7132c43bc914c1ed0c4d5f037163a986"
server_key_iv: "01f51796e9b7ca273140ff3e7de32277"
server_key_version: "encryptionkey@0"
I20260928 17:51:54.129024 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:54.146204 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:54.151506 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:54.151678 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:54.156157 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40795
I20260928 17:51:54.156203 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40795 every 8 connection(s)
I20260928 17:51:54.156575 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:54.157450 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:54.159766 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce: Bootstrap starting.
I20260928 17:51:54.160441 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:54.160732 3371 log.cc:824] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:54.161377 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce: No bootstrap required, opened a new log
I20260928 17:51:54.162962 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } }
I20260928 17:51:54.163123 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:54.163168 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26cc65660494007b06f22ec678655ce, State: Initialized, Role: FOLLOWER
I20260928 17:51:54.163316 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } }
I20260928 17:51:54.163395 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:54.163437 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:54.163491 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:54.164361 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } }
I20260928 17:51:54.164484 3371 leader_election.cc:302] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a26cc65660494007b06f22ec678655ce; no voters:
I20260928 17:51:54.164659 3371 leader_election.cc:288] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:54.164700 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:54.164873 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [term 1 LEADER]: Becoming Leader. State: Replica: a26cc65660494007b06f22ec678655ce, State: Running, Role: LEADER
I20260928 17:51:54.164925 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:54.165004 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } }
I20260928 17:51:54.165414 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a26cc65660494007b06f22ec678655ce. Latest consensus state: current_term: 1 leader_uuid: "a26cc65660494007b06f22ec678655ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } }
I20260928 17:51:54.165517 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:54.165527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3303
I20260928 17:51:54.165616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:54.165766 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a26cc65660494007b06f22ec678655ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26cc65660494007b06f22ec678655ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40795 } } }
I20260928 17:51:54.165802 392 external_mini_cluster.cc:1428] Setting key 5b18ee11e33eebc7266775295b4983ac
I20260928 17:51:54.165875 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:54.166059 3383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:54.166674 3383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:54.168371 3383 catalog_manager.cc:1347] Generated new cluster ID: 0689c6e88a434b229316c15cd4107a5c
I20260928 17:51:54.168426 3383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:54 dist-test-slave-zqlt krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:54.175781 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:54.166369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.98.62:40795)
0928 17:51:54.166562 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:54.166572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:54.166606 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:51:54.166727 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:54.166734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:54.166799 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:54.166939 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:54.167678 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:54.168914 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:54.169671 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:54.170105 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:54.170233 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:54.172525 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:54.172529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:54.172530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:54.172555 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:54.174264 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:54.174657 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:54.174660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:54.174660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:54.174748 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:54.174956 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:54.174957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:54.174957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:54.175192 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:54.175363 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:54.175435 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:51:54.175575 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 17:51:54.176707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:51:54.179621 3383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:54.180321 3383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:54.190006 3383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a26cc65660494007b06f22ec678655ce: Generated new TSK 0
I20260928 17:51:54.190230 3383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:51:54.197245 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:54.193439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.0.98.62:40795)
0928 17:51:54.193503 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:54.193505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:54.193586 (+ 81us) server_negotiation.cc:407] Connection header received
0928 17:51:54.193652 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:54.193653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:54.193665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:54.193742 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:54.194098 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:54.194919 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:54.195644 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:54.195977 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:54.196049 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:54.196761 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:54.196762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:54.196762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:51:54.196995 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:54.197015 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:51:54.197023 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:54.197082 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:54.197137 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 17:51:54.197162 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:51:54.199687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3303
2026-09-28T17:51:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:54 dist-test-slave-zqlt krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:54 dist-test-slave-zqlt krb5kdc[3404](info): set up 2 sockets
Sep 28 17:51:54 dist-test-slave-zqlt krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260928 17:51:55.240163 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:55.358458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:55.437991 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:55.438242 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:55.438282 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:55.439491 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:55.439558 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:55.439581 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:55.439604 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:55.439632 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:55.441190 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:55.441468 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:55.441702 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:55.446714 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.446715 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.446743 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.446748 3417 server_base.cc:1034] running on GCE node
I20260928 17:51:55.447184 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:55.447486 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:55.448637 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790617915448627 us; error 30 us; skew 500 ppm
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:55.451131 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:55.451589 3417 webserver.cc:460] Webserver started at http://127.0.98.62:44899/ using document root <none> and password file <none>
I20260928 17:51:55.451841 3417 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:55.451911 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:55.452062 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:55.453241 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "412833c568984832afa69642d84c74f7"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "be5b65edf1573af14c144a9f3ee8a88e"
server_key_iv: "0bf573e52b94fdd7db24178a5e77a9f4"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.453620 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "412833c568984832afa69642d84c74f7"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "be5b65edf1573af14c144a9f3ee8a88e"
server_key_iv: "0bf573e52b94fdd7db24178a5e77a9f4"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.454919 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:55.455626 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.455783 3417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:55.455889 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "412833c568984832afa69642d84c74f7"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "be5b65edf1573af14c144a9f3ee8a88e"
server_key_iv: "0bf573e52b94fdd7db24178a5e77a9f4"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.455967 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:55.480057 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:55.481973 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:55.482142 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:55.486359 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37493
I20260928 17:51:55.486400 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37493 every 8 connection(s)
I20260928 17:51:55.486760 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:55.487847 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:55.490343 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7: Bootstrap starting.
I20260928 17:51:55.490908 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:55.491145 3485 log.cc:824] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:55.491855 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7: No bootstrap required, opened a new log
I20260928 17:51:55.493436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3417
I20260928 17:51:55.493518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:55.493710 392 external_mini_cluster.cc:1428] Setting key 94714fc7db7d10db663e60b514c282a4
I20260928 17:51:55.493577 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260928 17:51:55.493806 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:55.494268 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412833c568984832afa69642d84c74f7, State: Initialized, Role: FOLLOWER
I20260928 17:51:55.494517 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260928 17:51:55.494632 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:55.494683 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:55.494745 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:55.495720 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260928 17:51:55.495849 3485 leader_election.cc:302] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412833c568984832afa69642d84c74f7; no voters:
I20260928 17:51:55.496052 3485 leader_election.cc:288] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:55.496070 3490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:55.496223 3490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [term 1 LEADER]: Becoming Leader. State: Replica: 412833c568984832afa69642d84c74f7, State: Running, Role: LEADER
I20260928 17:51:55.496356 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:55.496382 3490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260928 17:51:55.496752 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412833c568984832afa69642d84c74f7. Latest consensus state: current_term: 1 leader_uuid: "412833c568984832afa69642d84c74f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } } }
I20260928 17:51:55.497025 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:55.496920 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412833c568984832afa69642d84c74f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412833c568984832afa69642d84c74f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } } }
I20260928 17:51:55.497507 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:55.497520 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:55.498099 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:55.500032 3499 catalog_manager.cc:1347] Generated new cluster ID: 304ff94723954e9bbc81134b62026a5e
I20260928 17:51:55.500108 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:55.506007 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:55.506021 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.494227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.0.98.62:37493)
0928 17:51:55.494405 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:55.494425 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:55.494451 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:51:55.494580 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:55.494585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:55.494628 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:55.494780 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:55.495472 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.496711 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.497841 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.498523 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.498946 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.501699 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:55.501703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:55.501704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:55.501717 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:55.503791 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.504295 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.504299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.504299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.504427 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.504666 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.504667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.504667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.504968 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:55.505137 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:55.505649 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 17:51:55.505801 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 17:51:55.506726 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:55.510857 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412833c568984832afa69642d84c74f7: Generated new TSK 0
I20260928 17:51:55.511049 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:55.539800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:55.623100 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:55.623301 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:55.623325 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:55.624502 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:55.624544 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:55.624570 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:55.625973 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:55.626195 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:55.626430 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:55.631321 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.631314 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.631314 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.631808 3511 server_base.cc:1034] running on GCE node
I20260928 17:51:55.632017 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:55.632273 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:55.633440 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790617915633426 us; error 29 us; skew 500 ppm
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:55.635946 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:55.636466 3511 webserver.cc:460] Webserver started at http://127.0.98.1:32997/ using document root <none> and password file <none>
I20260928 17:51:55.636688 3511 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:55.636758 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:55.636894 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:55.637955 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4e43226507e45b3a828127aca6ba7b0"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "0cc2f515af81ee8b41f4c200e285a62d"
server_key_iv: "9a3ef82189682183b2fcd8041ca2ca63"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.638322 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4e43226507e45b3a828127aca6ba7b0"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "0cc2f515af81ee8b41f4c200e285a62d"
server_key_iv: "9a3ef82189682183b2fcd8041ca2ca63"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.639624 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:55.640430 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.640609 3511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:55.640707 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "e4e43226507e45b3a828127aca6ba7b0"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "0cc2f515af81ee8b41f4c200e285a62d"
server_key_iv: "9a3ef82189682183b2fcd8041ca2ca63"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.640839 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:55.666071 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:55.668265 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:55.668427 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:55.668700 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:55.669056 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:55.669091 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.669116 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:55.669163 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.675238 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39273
I20260928 17:51:55.675287 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39273 every 8 connection(s)
I20260928 17:51:55.675625 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:55.685190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3511
I20260928 17:51:55.685338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:55.685580 392 external_mini_cluster.cc:1428] Setting key 26e8df3f85abc4a16bdee82ac8af8c07
I20260928 17:51:55.685783 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.676408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60189 (local address 127.0.98.62:37493)
0928 17:51:55.676471 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:55.676473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:55.676676 (+ 203us) server_negotiation.cc:407] Connection header received
0928 17:51:55.677237 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:55.677238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:55.677255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:55.677307 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:55.678307 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.679095 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.680291 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.680510 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.680602 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.682928 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:55.682930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:55.682930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:55.682938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:55.684285 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.684703 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.684704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.684705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.684772 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.685053 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.685054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.685054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.685174 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:55.685250 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:55.685605 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 17:51:55.685691 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:51:55.686239 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.676394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.1:60189)
0928 17:51:55.676551 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:55.676589 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:55.677118 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:55.677386 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:55.677390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:55.677989 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:55.678195 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.678200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.679298 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.679299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.680200 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.680202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.680613 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.680614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.680753 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.681287 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:55.681297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:55.682824 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:55.684383 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.684391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.684393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.684622 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.684844 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.684845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.684845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.684972 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.685242 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:55.685248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:55.685443 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 17:51:55.685569 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 17:51:55.685739 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260928 17:51:55.686723 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260928 17:51:55.686836 3640 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:55.687064 3640 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
I20260928 17:51:55.687587 3449 ts_manager.cc:194] Registered new tserver with Master: e4e43226507e45b3a828127aca6ba7b0 (127.0.98.1:39273)
I20260928 17:51:55.688621 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60189
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:55.710409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:55.791702 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:55.791925 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:55.791951 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:55.793208 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:55.793255 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:55.793334 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:55.794889 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:55.795172 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:55.795411 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:55.800578 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.800709 3645 server_base.cc:1034] running on GCE node
W20260928 17:51:55.800631 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.800567 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.801011 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:55.801261 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:55.802434 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790617915802423 us; error 38 us; skew 500 ppm
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:55.804783 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:55.805253 3645 webserver.cc:460] Webserver started at http://127.0.98.2:36457/ using document root <none> and password file <none>
I20260928 17:51:55.805492 3645 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:55.805559 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:55.805706 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:55.806751 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "05daace363364a4e87464b68e1975aa3"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "934b0927acfecb2fc60972b5d00d8bdb"
server_key_iv: "68bff8b12d11d19370335ad4ece67f20"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.807106 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05daace363364a4e87464b68e1975aa3"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "934b0927acfecb2fc60972b5d00d8bdb"
server_key_iv: "68bff8b12d11d19370335ad4ece67f20"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.808480 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 17:51:55.809175 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.809340 3645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:55.809435 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "05daace363364a4e87464b68e1975aa3"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "934b0927acfecb2fc60972b5d00d8bdb"
server_key_iv: "68bff8b12d11d19370335ad4ece67f20"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.809511 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:55.815814 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:55.817633 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:55.817795 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:55.818022 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:55.818320 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:55.818382 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.818423 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:55.818451 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.824225 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36489
I20260928 17:51:55.824297 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36489 every 8 connection(s)
I20260928 17:51:55.824640 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:55.825481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3645
I20260928 17:51:55.825598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:55.825802 392 external_mini_cluster.cc:1428] Setting key b961230d86d4e105ec23589ffa27a1f1
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:51:55.835407 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.825440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44723 (local address 127.0.98.62:37493)
0928 17:51:55.825486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:55.825488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:55.825743 (+ 255us) server_negotiation.cc:407] Connection header received
0928 17:51:55.826482 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:55.826483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:55.826495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:55.826552 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:55.827506 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.828265 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.829567 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.829915 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.829998 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.832342 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:55.832344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:55.832345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:55.832353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:55.833959 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.834458 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.834460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.834460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.834523 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.834786 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.834787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.834787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.834900 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:55.834953 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:55.835246 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 17:51:55.835332 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:51:55.835563 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.825427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.2:44723)
0928 17:51:55.825604 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:55.825651 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:55.826375 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:55.826625 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:55.826629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:55.827212 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:55.827404 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.827410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.828359 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.828360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.829462 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.829464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.830008 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.830009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.830183 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.830782 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:55.830791 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:55.832244 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:55.834062 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.834068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.834070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.834379 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.834608 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.834609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.834609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.834721 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.834959 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:55.834964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:55.835117 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 17:51:55.835205 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:51:55.835334 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260928 17:51:55.836215 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260928 17:51:55.836321 3774 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:55.836552 3774 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 17:51:55.837007 3449 ts_manager.cc:194] Registered new tserver with Master: 05daace363364a4e87464b68e1975aa3 (127.0.98.2:36489)
I20260928 17:51:55.837396 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:55.850666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:55.930936 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:55.931190 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:55.931227 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:55.932452 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:55.932520 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:55.932565 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:55.933976 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:55.934242 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:55.934473 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:55.939329 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:55.939327 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.939405 3779 server_base.cc:1034] running on GCE node
W20260928 17:51:55.939329 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:55.939771 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:55.939968 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:55.941141 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790617915941139 us; error 36 us; skew 500 ppm
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:55.943646 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:55.944166 3779 webserver.cc:460] Webserver started at http://127.0.98.3:39405/ using document root <none> and password file <none>
I20260928 17:51:55.944413 3779 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:55.944473 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:55.944621 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:55.945663 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c66ae1a9f1a43c4bbaf026021dce741"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "11a237009e306cbd980456c71f5c91f9"
server_key_iv: "16b09952fe0b0e037cbded8d75b6661a"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.946025 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c66ae1a9f1a43c4bbaf026021dce741"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "11a237009e306cbd980456c71f5c91f9"
server_key_iv: "16b09952fe0b0e037cbded8d75b6661a"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.947419 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:55.948218 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.948379 3779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:55.948468 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "9c66ae1a9f1a43c4bbaf026021dce741"
format_stamp: "Formatted at 2026-09-28 17:51:55 on dist-test-slave-zqlt"
server_key: "11a237009e306cbd980456c71f5c91f9"
server_key_iv: "16b09952fe0b0e037cbded8d75b6661a"
server_key_version: "encryptionkey@0"
I20260928 17:51:55.948525 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:55.973592 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:55.975636 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:55.975747 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:55.975951 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:55.976279 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:55.976311 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.976333 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:55.976346 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:55.981556 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34849
I20260928 17:51:55.981633 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34849 every 8 connection(s)
I20260928 17:51:55.981961 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:55.985328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3779
I20260928 17:51:55.985435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:55.985625 392 external_mini_cluster.cc:1428] Setting key 3b881d2ab41a4697b22e7ced3576bbd3
Sep 28 17:51:55 dist-test-slave-zqlt krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:55.991091 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.982742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57777 (local address 127.0.98.62:37493)
0928 17:51:55.982824 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:55.982826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:55.983024 (+ 198us) server_negotiation.cc:407] Connection header received
0928 17:51:55.983539 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:55.983541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:55.983552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:55.983597 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:55.984549 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.985040 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.986336 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.986628 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.986689 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.988796 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:55.988797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:55.988797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:55.988803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:55.990032 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.990369 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.990369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.990370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.990417 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:55.990613 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:55.990614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:55.990614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:55.990707 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:55.990754 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:55.990973 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 17:51:55.991030 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:51:55.991233 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:55.982717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.3:57777)
0928 17:51:55.982871 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:55.982921 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:55.983434 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:55.983663 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:55.983668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:55.984253 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:55.984447 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.984452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.985147 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.985148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.986253 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:55.986255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:55.986708 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:55.986709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:55.986809 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:55.987307 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:55.987316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:55.988717 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:55.990100 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.990105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.990107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.990305 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.990472 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:55.990473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:55.990473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:55.990561 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:55.990758 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:55.990762 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:55.990863 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 17:51:55.990943 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:51:55.991053 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 17:51:55.991842 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260928 17:51:55.991916 3908 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:55.992120 3908 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
I20260928 17:51:55.992496 3449 ts_manager.cc:194] Registered new tserver with Master: 9c66ae1a9f1a43c4bbaf026021dce741 (127.0.98.3:34849)
I20260928 17:51:55.992869 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57777
I20260928 17:51:55.997305 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:51:56 dist-test-slave-zqlt krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617916, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 17:51:56 dist-test-slave-zqlt krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617916, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:56.019515 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:56.012177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.0.98.62:37493)
0928 17:51:56.012249 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:56.012252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:56.012360 (+ 108us) server_negotiation.cc:407] Connection header received
0928 17:51:56.012448 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:56.012449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:56.012466 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:56.012533 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:56.013075 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:56.013634 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:56.014406 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:56.014594 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:56.014676 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:56.016858 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:56.016860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:56.016860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:56.016867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:56.018262 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:56.018652 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:56.018654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:56.018654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:56.018734 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:56.018954 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:56.018955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:56.018956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:56.019102 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:56.019213 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:56.019332 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 17:51:56.019407 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 17:51:56.019762 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32784
I20260928 17:51:56.020256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3511
I20260928 17:51:56.024972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3645
I20260928 17:51:56.029062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3779
I20260928 17:51:56.032967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3417
2026-09-28T17:51:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1834 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:56 dist-test-slave-zqlt krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:56 dist-test-slave-zqlt krb5kdc[3920](info): set up 2 sockets
Sep 28 17:51:56 dist-test-slave-zqlt krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260928 17:51:57.080030 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:57.210192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:57.290889 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:57.291080 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:57.291102 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:57.292215 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:57.292249 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:57.292263 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:57.292275 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:57.292287 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:57.293632 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:57.293803 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:57.293962 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:57.298800 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:57.298954 3933 server_base.cc:1034] running on GCE node
W20260928 17:51:57.298770 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.298780 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:57.299309 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:57.299631 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:57.300781 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790617917300768 us; error 25 us; skew 500 ppm
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:57.303138 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:57.303614 3933 webserver.cc:460] Webserver started at http://127.0.98.62:33473/ using document root <none> and password file <none>
I20260928 17:51:57.303845 3933 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:57.303912 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:57.304060 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:57.305171 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "e244862b39c546b7bb2386f196b50d17"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "c64d02748faf988b15fb245baaaae6c0"
server_key_iv: "f36ac6053c1bf9da960764ac783d0150"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.305536 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "e244862b39c546b7bb2386f196b50d17"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "c64d02748faf988b15fb245baaaae6c0"
server_key_iv: "f36ac6053c1bf9da960764ac783d0150"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.306813 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:51:57.307519 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.307691 3933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:57.307788 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "e244862b39c546b7bb2386f196b50d17"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "c64d02748faf988b15fb245baaaae6c0"
server_key_iv: "f36ac6053c1bf9da960764ac783d0150"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.307874 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:57.318467 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:57.320436 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:57.320596 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:57.324638 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42599
I20260928 17:51:57.324661 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42599 every 8 connection(s)
I20260928 17:51:57.325019 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:57.325865 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:57.328265 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17: Bootstrap starting.
I20260928 17:51:57.328836 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:57.329084 4001 log.cc:824] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:57.329775 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17: No bootstrap required, opened a new log
I20260928 17:51:57.331350 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } }
I20260928 17:51:57.331532 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:57.331557 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e244862b39c546b7bb2386f196b50d17, State: Initialized, Role: FOLLOWER
I20260928 17:51:57.331668 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } }
I20260928 17:51:57.331756 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:57.331791 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:57.331844 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:57.332465 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } }
I20260928 17:51:57.332594 4001 leader_election.cc:302] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e244862b39c546b7bb2386f196b50d17; no voters:
I20260928 17:51:57.332798 4001 leader_election.cc:288] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:57.332913 4004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:57.333109 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:57.333316 4004 raft_consensus.cc:686] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [term 1 LEADER]: Becoming Leader. State: Replica: e244862b39c546b7bb2386f196b50d17, State: Running, Role: LEADER
I20260928 17:51:57.333412 4004 consensus_queue.cc:237] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } }
I20260928 17:51:57.333990 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e244862b39c546b7bb2386f196b50d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } } }
I20260928 17:51:57.334019 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e244862b39c546b7bb2386f196b50d17. Latest consensus state: current_term: 1 leader_uuid: "e244862b39c546b7bb2386f196b50d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e244862b39c546b7bb2386f196b50d17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42599 } } }
I20260928 17:51:57.334089 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:57.334101 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:57.334496 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:57.334904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 3933
I20260928 17:51:57.334992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:57.335029 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:57.335192 392 external_mini_cluster.cc:1428] Setting key ec67285ea585b2a13fd10e718080ccea
I20260928 17:51:57.336850 4013 catalog_manager.cc:1347] Generated new cluster ID: 1d3ab27221da4d4db35c8ecd29bf79b1
I20260928 17:51:57.336905 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:57.347574 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.335666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.98.62:42599)
0928 17:51:57.335858 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.335869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.335901 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:51:57.336011 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.336016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.336060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.336235 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.337171 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.338573 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.339589 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.340261 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.340364 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.343590 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.343594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.343594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.343607 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.345688 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.346105 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.346108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.346108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.346224 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.346461 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.346462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.346462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.346815 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.346966 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.347095 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.347328 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260928 17:51:57.351979 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:57.352522 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:57.371004 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e244862b39c546b7bb2386f196b50d17: Generated new TSK 0
I20260928 17:51:57.371263 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:57.398856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:57.480347 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:57.480604 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:57.480643 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:57.481823 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:57.481895 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:57.481940 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:57.483456 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:57.483729 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:57.483965 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:57.488776 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.488768 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.488782 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:57.489144 4027 server_base.cc:1034] running on GCE node
I20260928 17:51:57.489351 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:57.489548 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:57.490680 4027 hybrid_clock.cc:648] HybridClock initialized: now 1790617917490667 us; error 29 us; skew 500 ppm
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:57.493149 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:57.493613 4027 webserver.cc:460] Webserver started at http://127.0.98.1:36905/ using document root <none> and password file <none>
I20260928 17:51:57.493839 4027 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:57.493911 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:57.494062 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:57.495059 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "948d41175602432693348317020ebdfa"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "266985344ecf09aea36f9454a23f4ac9"
server_key_iv: "ae36d2408e5f37510e87d459335e3602"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.495422 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "948d41175602432693348317020ebdfa"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "266985344ecf09aea36f9454a23f4ac9"
server_key_iv: "ae36d2408e5f37510e87d459335e3602"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.496757 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:51:57.497421 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.497588 4027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:57.497681 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "948d41175602432693348317020ebdfa"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "266985344ecf09aea36f9454a23f4ac9"
server_key_iv: "ae36d2408e5f37510e87d459335e3602"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.497761 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:57.508565 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:57.510228 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:57.510385 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:57.510605 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:57.510891 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:57.510946 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.510985 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:57.511013 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.516772 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260928 17:51:57.516817 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260928 17:51:57.517110 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:57.523906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4027
I20260928 17:51:57.524015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:57.524237 392 external_mini_cluster.cc:1428] Setting key 0c43af1e64e523848945be7e881560e3
I20260928 17:51:57.526306 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.517794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56281 (local address 127.0.98.62:42599)
0928 17:51:57.517872 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.517875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.518034 (+ 159us) server_negotiation.cc:407] Connection header received
0928 17:51:57.518911 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.518913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.518923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.518970 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.519525 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.520019 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.521010 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.521209 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.521262 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.523262 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.523263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.523264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.523271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.524977 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.525354 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.525356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.525356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.525416 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.525670 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.525671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.525671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.525786 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.525858 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.526129 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.526216 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 17:51:57.526482 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.517744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42599 (local address 127.0.98.1:56281)
0928 17:51:57.517919 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:57.517951 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:57.518814 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:57.519036 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:57.519039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:57.519276 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:57.519454 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.519458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.520136 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.520137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.520935 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.520937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.521280 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.521281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.521372 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.521893 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:57.521900 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:57.523181 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:57.525055 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.525061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.525063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.525273 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.525490 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.525491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.525491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.525593 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.525870 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:57.525876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:57.526002 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:51:57.526108 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 17:51:57.526254 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260928 17:51:57.527187 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42599
I20260928 17:51:57.527287 4156 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:57.527451 4156 heartbeater.cc:511] Master 127.0.98.62:42599 requested a full tablet report, sending...
I20260928 17:51:57.528020 3965 ts_manager.cc:194] Registered new tserver with Master: 948d41175602432693348317020ebdfa (127.0.98.1:37025)
I20260928 17:51:57.528694 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56281
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:57.546773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:57.629248 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:57.629494 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:57.629520 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:57.630740 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:57.630786 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:57.630815 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:57.632349 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:57.632613 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:57.632845 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:57.637998 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.638005 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.638005 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:57.638185 4161 server_base.cc:1034] running on GCE node
I20260928 17:51:57.638476 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:57.638688 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:57.639832 4161 hybrid_clock.cc:648] HybridClock initialized: now 1790617917639821 us; error 32 us; skew 500 ppm
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:57.642437 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:57.642877 4161 webserver.cc:460] Webserver started at http://127.0.98.2:39297/ using document root <none> and password file <none>
I20260928 17:51:57.643102 4161 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:57.643148 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:57.643302 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:57.644446 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "bb6e0fbb55674ca19e91af768f8b964a"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "2f435eed11568e6afebabd2705d2d835"
server_key_iv: "0ec2fcf03af9b1c7128def81b5fdb785"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.644804 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bb6e0fbb55674ca19e91af768f8b964a"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "2f435eed11568e6afebabd2705d2d835"
server_key_iv: "0ec2fcf03af9b1c7128def81b5fdb785"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.646086 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:51:57.646821 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.647028 4161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:57.647125 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "bb6e0fbb55674ca19e91af768f8b964a"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "2f435eed11568e6afebabd2705d2d835"
server_key_iv: "0ec2fcf03af9b1c7128def81b5fdb785"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.647220 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:57.658023 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:57.659766 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:57.659925 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:57.660215 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:57.660528 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:57.660583 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.660624 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:57.660650 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.666326 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35477
I20260928 17:51:57.666366 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35477 every 8 connection(s)
I20260928 17:51:57.666671 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:57.671907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4161
I20260928 17:51:57.672019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:57.672254 392 external_mini_cluster.cc:1428] Setting key 056974c73b7ca440d490970d2ff8f21f
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:57.676151 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.667360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51833 (local address 127.0.98.62:42599)
0928 17:51:57.667425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.667427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.667629 (+ 202us) server_negotiation.cc:407] Connection header received
0928 17:51:57.668651 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.668653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.668664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.668731 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.669298 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.669783 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.670702 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.670876 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.670924 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.673103 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.673105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.673105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.673113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.674626 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.675112 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.675112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.675113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.675160 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.675447 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.675448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.675448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.675588 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.675660 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.675898 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.675991 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:51:57.676189 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.667341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42599 (local address 127.0.98.2:51833)
0928 17:51:57.667498 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:57.667540 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:57.668546 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:57.668785 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:57.668789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:57.669040 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:57.669226 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.669230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.669860 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.669861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.670634 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.670636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.670944 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.670944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.671040 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.671499 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:57.671508 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:57.673014 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:57.674705 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.674711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.674714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.675036 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.675227 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.675228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.675229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.675365 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.675655 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:57.675659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:57.675767 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 17:51:57.675849 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 17:51:57.675972 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260928 17:51:57.676657 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42599
I20260928 17:51:57.676760 4290 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:57.676924 4290 heartbeater.cc:511] Master 127.0.98.62:42599 requested a full tablet report, sending...
I20260928 17:51:57.677292 3965 ts_manager.cc:194] Registered new tserver with Master: bb6e0fbb55674ca19e91af768f8b964a (127.0.98.2:35477)
I20260928 17:51:57.677824 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51833
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:57.696239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:57.777001 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:57.777211 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:57.777235 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:57.778363 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:57.778398 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:57.778424 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:57.779857 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:57.780057 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:57.780320 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:57.785835 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.785867 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:57.785835 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:57.786134 4295 server_base.cc:1034] running on GCE node
I20260928 17:51:57.786381 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:57.786651 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:57.787824 4295 hybrid_clock.cc:648] HybridClock initialized: now 1790617917787816 us; error 43 us; skew 500 ppm
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:57.790474 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:57.790875 4295 webserver.cc:460] Webserver started at http://127.0.98.3:39479/ using document root <none> and password file <none>
I20260928 17:51:57.791072 4295 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:57.791112 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:57.791198 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:57.792259 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "b84104c1d4d04e5489bb1c4b61bd4953"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "e61d8be537ca208229924aaf289e023b"
server_key_iv: "ffb897ed653fe9bf96e16f4b3413c6bc"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.792569 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b84104c1d4d04e5489bb1c4b61bd4953"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "e61d8be537ca208229924aaf289e023b"
server_key_iv: "ffb897ed653fe9bf96e16f4b3413c6bc"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.793792 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:57.794456 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.794607 4295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.794694 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "b84104c1d4d04e5489bb1c4b61bd4953"
format_stamp: "Formatted at 2026-09-28 17:51:57 on dist-test-slave-zqlt"
server_key: "e61d8be537ca208229924aaf289e023b"
server_key_iv: "ffb897ed653fe9bf96e16f4b3413c6bc"
server_key_version: "encryptionkey@0"
I20260928 17:51:57.794746 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:57.817391 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:57.819240 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:57.819355 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:57.819545 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:57.819882 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:57.819916 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.819938 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:57.819988 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:57.825939 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38387
I20260928 17:51:57.825980 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38387 every 8 connection(s)
I20260928 17:51:57.826314 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:57.831089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4295
I20260928 17:51:57.831205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:57.831764 392 external_mini_cluster.cc:1428] Setting key cc37a1cf1de00aa803b8608502b42811
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:57.835903 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.827176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60997 (local address 127.0.98.62:42599)
0928 17:51:57.827247 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.827248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.827456 (+ 208us) server_negotiation.cc:407] Connection header received
0928 17:51:57.828486 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.828487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.828497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.828554 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.829152 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.829664 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.830615 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.830788 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.830840 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.833391 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.833393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.833393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.833399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.834846 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.835196 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.835196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.835197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.835245 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.835417 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.835418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.835418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.835514 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.835562 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.835774 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.835840 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:51:57.836093 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.827140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42599 (local address 127.0.98.3:60997)
0928 17:51:57.827303 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:57.827346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:57.828391 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:57.828620 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:57.828623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:57.828885 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:57.829076 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.829080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.829735 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.829736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.830542 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:57.830544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.830858 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.830859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:57.831019 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.831624 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:57.831635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:57.833309 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:57.834916 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.834921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.834923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.835135 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.835285 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:57.835285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:57.835286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:57.835369 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:57.835552 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:57.835555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:57.835663 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 17:51:57.835739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:51:57.835860 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260928 17:51:57.836596 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42599
I20260928 17:51:57.836678 4424 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:57.836819 4424 heartbeater.cc:511] Master 127.0.98.62:42599 requested a full tablet report, sending...
I20260928 17:51:57.837148 3965 ts_manager.cc:194] Registered new tserver with Master: b84104c1d4d04e5489bb1c4b61bd4953 (127.0.98.3:38387)
I20260928 17:51:57.837495 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60997
I20260928 17:51:57.843842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:51:57.866940 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.859367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.1:37025)
0928 17:51:57.859534 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.859539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.859550 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:51:57.859577 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.859579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.859594 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.859661 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.860307 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.861029 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.861736 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.862057 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.862110 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.864350 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.864368 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.864368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.864375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.865979 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.866268 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.866271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.866271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.866326 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.866467 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.866467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.866468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.866590 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.866697 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.866751 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.866813 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:57.874405 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:57.868241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.62:42599)
0928 17:51:57.868315 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:57.868317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:57.868333 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:51:57.868399 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:57.868400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:57.868409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:57.868460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:57.868895 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.869366 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.869998 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:57.870172 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:57.870221 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:57.872312 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:57.872313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:57.872313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:57.872320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:57.873404 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.873724 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.873724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.873725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.873773 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:57.873948 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:57.873948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:57.873948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:57.874089 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:57.874161 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:57.874245 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:51:57.874302 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 17:51:57.874610 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50982
I20260928 17:51:57.874944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4027
I20260928 17:51:57.879650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4161
I20260928 17:51:57.883810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4295
I20260928 17:51:57.887857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 3933
2026-09-28T17:51:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1854 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[4436](info): set up 2 sockets
Sep 28 17:51:57 dist-test-slave-zqlt krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260928 17:51:58.972671 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:51:59.091957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:59.173408 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:59.173617 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:59.173640 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:59.174731 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:59.174765 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:59.174779 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:59.174793 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:59.174806 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:59.176308 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:59.176499 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:59.176664 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:59.181367 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.181457 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.181370 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:59.181540 4449 server_base.cc:1034] running on GCE node
I20260928 17:51:59.181885 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:59.182207 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:59.183386 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790617919183370 us; error 28 us; skew 500 ppm
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:59.191205 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:51:59.191568 4449 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:59.191642 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:59.191798 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:59.192960 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "3ab91ee5a69237b73287b435d65b457e"
server_key_iv: "2282b6e23e85da41246770d10623b5e5"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.193331 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "3ab91ee5a69237b73287b435d65b457e"
server_key_iv: "2282b6e23e85da41246770d10623b5e5"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.194654 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:51:59.195387 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.195551 4449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:59.195655 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "3ab91ee5a69237b73287b435d65b457e"
server_key_iv: "2282b6e23e85da41246770d10623b5e5"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.195729 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:59.208205 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:59.210573 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:59.210739 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:59.214679 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42201
I20260928 17:51:59.214701 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42201 every 8 connection(s)
I20260928 17:51:59.214999 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:59.215917 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:59.216794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4449
I20260928 17:51:59.216917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:59.217088 392 external_mini_cluster.cc:1428] Setting key 109334cf8cb81d9d18ad9e1ffc716f54
I20260928 17:51:59.219084 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: Bootstrap starting.
I20260928 17:51:59.219794 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:59.220120 4516 log.cc:824] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:59.220816 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: No bootstrap required, opened a new log
I20260928 17:51:59.222888 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260928 17:51:59.223022 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:59.223044 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec36ce2bc7d4de0943a54e09cc9d7df, State: Initialized, Role: FOLLOWER
I20260928 17:51:59.223178 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260928 17:51:59.223258 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:59.223317 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:59.223367 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 0 FOLLOWER]: Advancing to term 1
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:59.223924 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260928 17:51:59.224035 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec36ce2bc7d4de0943a54e09cc9d7df; no voters:
I20260928 17:51:59.224270 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:59.224301 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:59.224409 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [term 1 LEADER]: Becoming Leader. State: Replica: 8ec36ce2bc7d4de0943a54e09cc9d7df, State: Running, Role: LEADER
I20260928 17:51:59.224509 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260928 17:51:59.224547 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:59.224822 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260928 17:51:59.224891 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:59.225102 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec36ce2bc7d4de0943a54e09cc9d7df. Latest consensus state: current_term: 1 leader_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec36ce2bc7d4de0943a54e09cc9d7df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260928 17:51:59.225229 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:59.225427 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:59.225960 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:59.227811 4527 catalog_manager.cc:1347] Generated new cluster ID: dd513a548200437ea08d314ace313354
I20260928 17:51:59.227881 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:51:59.228593 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.217589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.0.98.62:42201)
0928 17:51:59.217765 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.217775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.217816 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:51:59.218331 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.218340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.218406 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.218558 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.219121 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.220499 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.221308 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.221932 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.222040 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.224522 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.224527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.224527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:59.224541 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.226551 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.226947 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.226951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.226951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.227065 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.227327 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.227328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.227329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.227644 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.227786 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.228265 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.228397 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 17:51:59.250321 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:59.251046 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:59.255858 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: Generated new TSK 0
I20260928 17:51:59.256080 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:51:59.281054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:59.362071 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:59.362282 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:59.362306 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:59.363411 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:59.363446 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:59.363472 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:51:59.364948 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:59.365159 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:59.365343 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:59.370093 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.370118 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.370143 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:59.370343 4542 server_base.cc:1034] running on GCE node
I20260928 17:51:59.370561 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:59.370783 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:59.371951 4542 hybrid_clock.cc:648] HybridClock initialized: now 1790617919371944 us; error 37 us; skew 500 ppm
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:59.374352 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:51:59.374670 4542 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:59.374739 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:59.374886 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:59.375893 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce09e565dc37465293b88850d15f2670"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "75c9e61bdd99dd0efb6de23a6d0a127a"
server_key_iv: "28a31989fbd657ef7931447fe302c648"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.376269 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce09e565dc37465293b88850d15f2670"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "75c9e61bdd99dd0efb6de23a6d0a127a"
server_key_iv: "28a31989fbd657ef7931447fe302c648"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.377509 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:59.378196 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.378371 4542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:59.378465 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "ce09e565dc37465293b88850d15f2670"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "75c9e61bdd99dd0efb6de23a6d0a127a"
server_key_iv: "28a31989fbd657ef7931447fe302c648"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.378538 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:59.391093 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:59.392863 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:59.393016 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:59.393229 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:59.393667 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:59.393749 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.393788 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:59.393828 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.400727 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46723
I20260928 17:51:59.400811 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46723 every 8 connection(s)
I20260928 17:51:59.401096 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:51:59.405591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4542
I20260928 17:51:59.405721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:51:59.405920 392 external_mini_cluster.cc:1428] Setting key 5fe3cc31f7b3f724d147c81047203850
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:59.411388 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.401905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54693 (local address 127.0.98.62:42201)
0928 17:51:59.401955 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.401957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.402203 (+ 246us) server_negotiation.cc:407] Connection header received
0928 17:51:59.402762 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.402763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.402777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.402833 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.403808 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.404390 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.405434 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.405635 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.405721 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.408511 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.408513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.408513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:59.408521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.409937 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.410369 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.410370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.410371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.410437 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.410700 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.410701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.410701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.410821 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.410894 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.411186 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.411279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:51:59.411587 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.401861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.1:54693)
0928 17:51:59.402072 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.402110 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.402640 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.402914 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.402918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.403204 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:59.403702 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.403707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.404481 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.404482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.405316 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.405318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.405756 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.405758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.405923 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.406570 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:59.406607 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:59.408408 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:59.410040 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.410047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.410049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.410286 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.410505 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.410506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.410507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.410625 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.410896 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:59.410904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:59.411054 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.411168 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.411334 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260928 17:51:59.412156 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260928 17:51:59.412256 4670 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:59.412441 4670 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260928 17:51:59.413029 4480 ts_manager.cc:194] Registered new tserver with Master: ce09e565dc37465293b88850d15f2670 (127.0.98.1:46723)
I20260928 17:51:59.413657 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54693
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:51:59.430956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:59.512967 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:59.513180 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:59.513206 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:59.514282 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:59.514323 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:59.514351 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:51:59.515844 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:59.516036 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:59.516288 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:59.521361 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.521373 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.521373 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:59.521423 4675 server_base.cc:1034] running on GCE node
I20260928 17:51:59.521751 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:59.522004 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:59.523175 4675 hybrid_clock.cc:648] HybridClock initialized: now 1790617919523163 us; error 28 us; skew 500 ppm
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:59.525991 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:51:59.526330 4675 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:59.526401 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:59.526557 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:59.527614 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd81fdbd6b7f411baa86152e0f7c58e0"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "847f7c86c4849a33b9c045ac98e79ae0"
server_key_iv: "0e5681a98c6d5a9a09030020c560cc21"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.528002 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd81fdbd6b7f411baa86152e0f7c58e0"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "847f7c86c4849a33b9c045ac98e79ae0"
server_key_iv: "0e5681a98c6d5a9a09030020c560cc21"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.529300 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:51:59.530035 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.530202 4675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:51:59.530303 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "fd81fdbd6b7f411baa86152e0f7c58e0"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "847f7c86c4849a33b9c045ac98e79ae0"
server_key_iv: "0e5681a98c6d5a9a09030020c560cc21"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.530397 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:59.544790 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:59.546684 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:59.546864 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:59.547107 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:59.547438 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:59.547494 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.547537 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:59.547581 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.553341 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40405
I20260928 17:51:59.553419 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40405 every 8 connection(s)
I20260928 17:51:59.553694 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:51:59.555863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4675
I20260928 17:51:59.555967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:51:59.556164 392 external_mini_cluster.cc:1428] Setting key ae5556aceeaeb01993ea6f86b2cdb0ca
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:59.563632 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.554414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43855 (local address 127.0.98.62:42201)
0928 17:51:59.554482 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.554485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.554674 (+ 189us) server_negotiation.cc:407] Connection header received
0928 17:51:59.555135 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.555136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.555146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.555231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.556251 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.556758 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.557818 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.558129 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.558198 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.560702 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.560703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.560703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:59.560712 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.562154 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.562655 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.562656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.562656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.562707 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.562976 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.562976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.562976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.563076 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.563127 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.563505 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.563566 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:51:59.563905 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.554400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.2:43855)
0928 17:51:59.554553 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.554588 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.555049 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.555256 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.555259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.555517 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:59.556141 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.556148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.556876 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.556877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.557705 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.557707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.558225 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.558226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.558336 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.558874 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:59.558887 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:59.560607 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:59.562241 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.562248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.562250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.562571 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.562772 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.562772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.562773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.562909 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.563143 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:59.563148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:59.563353 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.563489 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.563674 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:51:59.564400 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260928 17:51:59.564495 4803 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:59.564613 4803 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260928 17:51:59.564949 4480 ts_manager.cc:194] Registered new tserver with Master: fd81fdbd6b7f411baa86152e0f7c58e0 (127.0.98.2:40405)
I20260928 17:51:59.565383 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:51:59.580360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:51:59.661636 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:59.661845 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:59.661868 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:59.663076 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:59.663148 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:59.663177 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:51:59.664736 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:59.664979 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:59.665217 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:59.670192 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.670214 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.670147 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:59.670481 4808 server_base.cc:1034] running on GCE node
I20260928 17:51:59.670711 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:59.670998 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:59.672204 4808 hybrid_clock.cc:648] HybridClock initialized: now 1790617919672201 us; error 62 us; skew 500 ppm
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:51:59.675043 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:51:59.675464 4808 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:59.675539 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:59.675688 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:59.676896 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "2293a0d8eec14d29b128e99902b17334"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "b6c01217aba3af35fa12ff70a13309ae"
server_key_iv: "1e64901f0f3858c197c502f4454a04b4"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.677289 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2293a0d8eec14d29b128e99902b17334"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "b6c01217aba3af35fa12ff70a13309ae"
server_key_iv: "1e64901f0f3858c197c502f4454a04b4"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.678730 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:51:59.679585 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.679806 4808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:51:59.679917 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "2293a0d8eec14d29b128e99902b17334"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "b6c01217aba3af35fa12ff70a13309ae"
server_key_iv: "1e64901f0f3858c197c502f4454a04b4"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.679997 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:59.690266 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:59.692726 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:59.692888 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:59.693123 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:51:59.693465 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:51:59.693518 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.693557 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:51:59.693584 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.699616 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45625
I20260928 17:51:59.699653 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45625 every 8 connection(s)
I20260928 17:51:59.699934 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:51:59.705056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4808
I20260928 17:51:59.705185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:51:59.705406 392 external_mini_cluster.cc:1428] Setting key 9cea383d8189851fd038d55a8b192384
Sep 28 17:51:59 dist-test-slave-zqlt krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:51:59.709144 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.700715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45211 (local address 127.0.98.62:42201)
0928 17:51:59.700778 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.700780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.700996 (+ 216us) server_negotiation.cc:407] Connection header received
0928 17:51:59.701455 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.701456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.701466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.701518 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.702428 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.702927 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.703868 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.704050 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.704186 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.706347 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.706349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.706349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:59.706357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.707744 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.708187 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.708188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.708189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.708249 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.708507 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.708508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.708508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.708623 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.708703 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.708968 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.709040 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:51:59.709337 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.700703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.3:45211)
0928 17:51:59.700878 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.700914 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.701375 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.701573 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.701577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.701853 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:51:59.702350 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.702354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.703004 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.703005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.703795 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.703797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.704162 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.704167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.704266 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.704755 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:51:59.704763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:51:59.706250 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:51:59.707857 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.707863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.707865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.708123 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.708317 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:51:59.708318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:51:59.708318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:51:59.708437 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:51:59.708695 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:51:59.708701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:51:59.708839 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.708949 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.709103 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260928 17:51:59.709831 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260928 17:51:59.709933 4936 heartbeater.cc:464] Registering TS with master...
I20260928 17:51:59.710047 4936 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260928 17:51:59.710402 4480 ts_manager.cc:194] Registered new tserver with Master: 2293a0d8eec14d29b128e99902b17334 (127.0.98.3:45625)
I20260928 17:51:59.710743 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45211
I20260928 17:51:59.717237 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:51:59.724436 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.718601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.0.98.62:42201)
0928 17:51:59.718661 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.718664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.718719 (+ 55us) server_negotiation.cc:407] Connection header received
0928 17:51:59.718773 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.718774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.718781 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.718836 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.719357 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.719832 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.720617 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.720831 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.720885 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.721952 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.721953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.721953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:51:59.721959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.723335 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.723652 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.723653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.723653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.723723 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:51:59.723884 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:51:59.723884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:51:59.723884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:51:59.724013 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.724135 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.724269 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.724355 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:51:59.726127 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:51:59.727797 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:51:59.734264 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46723 (local address 127.0.0.1:60076)
0928 17:51:59.730373 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.730414 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.730486 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.730795 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.730799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.730801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:59.730870 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.730872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.731790 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.731791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.733065 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.733068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.733962 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.733964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.734041 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734057 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.734124 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.734180 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":67,"threads_started":1}
I20260928 17:51:59.734429 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.3:45625)
0928 17:51:59.730665 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.730669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.730679 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:51:59.730717 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.730720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.730734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.730805 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:59.730996 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.732154 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.733462 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.734034 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.734109 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734176 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.734290 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.734335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":92,"threads_started":1}
I20260928 17:51:59.734622 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.0.98.1:46723)
0928 17:51:59.730455 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.730462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.730474 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:51:59.730579 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.730582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.730600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.730692 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:59.730955 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.731677 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.733148 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.733878 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.733952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734011 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.734494 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.734545 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 17:51:59.734669 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.98.2:40405)
0928 17:51:59.730554 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.730557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.730764 (+ 207us) server_negotiation.cc:407] Connection header received
0928 17:51:59.730811 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.730814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.730830 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.730902 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:59.731128 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.732198 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.733436 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.734133 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.734192 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734244 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.734516 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.734582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260928 17:51:59.734812 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40405 (local address 127.0.0.1:59720)
0928 17:51:59.730593 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.730600 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.730655 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.730991 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.730993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.730994 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:59.731049 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.731050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.732297 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.732298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.733339 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.733342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.734274 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.734275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.734334 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734337 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.734388 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.734699 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":2,"thread_start_us":36,"threads_started":1}
I20260928 17:51:59.735074 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.730073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45625 (local address 127.0.0.1:38162)
0928 17:51:59.730458 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.730468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.730509 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.730886 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.730887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.730889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:59.730931 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.730933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.732274 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.732275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.733373 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.733376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.734141 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.734142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.734209 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.734213 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.734274 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.734659 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":33,"spinlock_wait_cycles":896,"thread_start_us":63,"threads_started":1}
I20260928 17:51:59.735340 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 14abf3e9224b42cd9652f208de0dee63 (DEFAULT_TABLE table=test-table [id=3f5a35d1d26f433bbdb0331d55c1bb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:59.735428 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 14abf3e9224b42cd9652f208de0dee63 (DEFAULT_TABLE table=test-table [id=3f5a35d1d26f433bbdb0331d55c1bb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:59.735714 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14abf3e9224b42cd9652f208de0dee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:59.735714 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14abf3e9224b42cd9652f208de0dee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:59.735729 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 14abf3e9224b42cd9652f208de0dee63 (DEFAULT_TABLE table=test-table [id=3f5a35d1d26f433bbdb0331d55c1bb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:51:59.736063 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14abf3e9224b42cd9652f208de0dee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:59.738773 4950 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Bootstrap starting.
I20260928 17:51:59.738947 4952 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: Bootstrap starting.
I20260928 17:51:59.739579 4950 tablet_bootstrap.cc:654] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:59.739579 4952 tablet_bootstrap.cc:654] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:59.740244 4951 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Bootstrap starting.
I20260928 17:51:59.740249 4952 log.cc:824] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:59.740248 4950 log.cc:824] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:59.740895 4951 tablet_bootstrap.cc:654] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:59.741104 4950 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: No bootstrap required, opened a new log
I20260928 17:51:59.741185 4950 ts_tablet_manager.cc:1397] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 17:51:59.741302 4952 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: No bootstrap required, opened a new log
I20260928 17:51:59.741387 4952 ts_tablet_manager.cc:1397] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:59.741477 4951 log.cc:824] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:59.742308 4951 tablet_bootstrap.cc:492] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: No bootstrap required, opened a new log
I20260928 17:51:59.742376 4951 ts_tablet_manager.cc:1397] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:51:59.743036 4950 raft_consensus.cc:348] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.743189 4950 raft_consensus.cc:374] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:59.743233 4950 raft_consensus.cc:729] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2293a0d8eec14d29b128e99902b17334, State: Initialized, Role: FOLLOWER
I20260928 17:51:59.743360 4950 consensus_queue.cc:260] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.743379 4952 raft_consensus.cc:348] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.743552 4952 raft_consensus.cc:374] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:59.743595 4952 raft_consensus.cc:729] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce09e565dc37465293b88850d15f2670, State: Initialized, Role: FOLLOWER
I20260928 17:51:59.743650 4950 ts_tablet_manager.cc:1428] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 17:51:59.743778 4936 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260928 17:51:59.743710 4952 consensus_queue.cc:260] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.744280 4952 ts_tablet_manager.cc:1428] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:51:59.744405 4670 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260928 17:51:59.744395 4951 raft_consensus.cc:348] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.744539 4951 raft_consensus.cc:374] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:59.744580 4951 raft_consensus.cc:729] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd81fdbd6b7f411baa86152e0f7c58e0, State: Initialized, Role: FOLLOWER
I20260928 17:51:59.744700 4951 consensus_queue.cc:260] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.744973 4951 ts_tablet_manager.cc:1428] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 17:51:59.745010 4803 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260928 17:51:59.780916 4956 raft_consensus.cc:482] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:51:59.781025 4956 raft_consensus.cc:504] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.781450 4956 leader_election.cc:288] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd81fdbd6b7f411baa86152e0f7c58e0 (127.0.98.2:40405), ce09e565dc37465293b88850d15f2670 (127.0.98.1:46723)
I20260928 17:51:59.784416 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.781695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40497 (local address 127.0.98.2:40405)
0928 17:51:59.781736 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.781738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.781748 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:51:59.781820 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.781821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.781833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.781911 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:59.782070 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.782536 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.783590 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.784038 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.784139 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.784151 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.784316 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.784354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:51:59.784416 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.781608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40405 (local address 127.0.98.3:40497)
0928 17:51:59.781670 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.781682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.781723 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.781913 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.781914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.781916 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:59.781990 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.781992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.782618 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.782619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.783445 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.783447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.784137 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.784138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.784220 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.784222 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.784299 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.784343 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 17:51:59.784545 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.781626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46723 (local address 127.0.98.3:60425)
0928 17:51:59.781778 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.781805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 17:51:59.781867 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:51:59.782020 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:51:59.782021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:51:59.782022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:51:59.782092 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.782095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.782747 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.782748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.783530 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:51:59.783532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.784267 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.784268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:51:59.784343 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.784349 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:51:59.784429 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:51:59.784467 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 17:51:59.784533 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.781717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60425 (local address 127.0.98.1:46723)
0928 17:51:59.781779 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.781781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.781868 (+ 87us) server_negotiation.cc:407] Connection header received
0928 17:51:59.781932 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.781932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.781944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.782017 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:51:59.782167 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.782671 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.783619 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.784187 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.784267 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.784276 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.784438 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.784473 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:51:59.784771 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14abf3e9224b42cd9652f208de0dee63" candidate_uuid: "2293a0d8eec14d29b128e99902b17334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" is_pre_election: true
I20260928 17:51:59.784812 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14abf3e9224b42cd9652f208de0dee63" candidate_uuid: "2293a0d8eec14d29b128e99902b17334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce09e565dc37465293b88850d15f2670" is_pre_election: true
I20260928 17:51:59.784935 4757 raft_consensus.cc:2454] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2293a0d8eec14d29b128e99902b17334 in term 0.
I20260928 17:51:59.784957 4624 raft_consensus.cc:2454] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2293a0d8eec14d29b128e99902b17334 in term 0.
I20260928 17:51:59.785146 4826 leader_election.cc:302] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2293a0d8eec14d29b128e99902b17334, ce09e565dc37465293b88850d15f2670; no voters:
I20260928 17:51:59.785300 4956 raft_consensus.cc:2781] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:51:59.785355 4956 raft_consensus.cc:482] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:51:59.785378 4956 raft_consensus.cc:3037] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:59.786039 4956 raft_consensus.cc:504] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.786172 4956 leader_election.cc:288] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [CANDIDATE]: Term 1 election: Requested vote from peers fd81fdbd6b7f411baa86152e0f7c58e0 (127.0.98.2:40405), ce09e565dc37465293b88850d15f2670 (127.0.98.1:46723)
I20260928 17:51:59.786307 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14abf3e9224b42cd9652f208de0dee63" candidate_uuid: "2293a0d8eec14d29b128e99902b17334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0"
I20260928 17:51:59.786352 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14abf3e9224b42cd9652f208de0dee63" candidate_uuid: "2293a0d8eec14d29b128e99902b17334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce09e565dc37465293b88850d15f2670"
I20260928 17:51:59.786393 4757 raft_consensus.cc:3037] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:59.786455 4624 raft_consensus.cc:3037] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:59.787108 4757 raft_consensus.cc:2454] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2293a0d8eec14d29b128e99902b17334 in term 1.
I20260928 17:51:59.787204 4624 raft_consensus.cc:2454] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2293a0d8eec14d29b128e99902b17334 in term 1.
I20260928 17:51:59.787277 4825 leader_election.cc:302] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2293a0d8eec14d29b128e99902b17334, fd81fdbd6b7f411baa86152e0f7c58e0; no voters:
I20260928 17:51:59.787367 4956 raft_consensus.cc:2781] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:59.787523 4956 raft_consensus.cc:686] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 1 LEADER]: Becoming Leader. State: Replica: 2293a0d8eec14d29b128e99902b17334, State: Running, Role: LEADER
I20260928 17:51:59.787644 4956 consensus_queue.cc:237] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } }
I20260928 17:51:59.788597 4478 catalog_manager.cc:5495] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2293a0d8eec14d29b128e99902b17334 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2293a0d8eec14d29b128e99902b17334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2293a0d8eec14d29b128e99902b17334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 } health_report { overall_health: UNKNOWN } } }
W20260928 17:51:59.804261 4804 tablet.cc:2369] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:51:59.807964 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.805522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.0.98.3:45625)
0928 17:51:59.805587 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.805589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.805600 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:51:59.805642 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.805642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.805671 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.805750 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:51:59.805924 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.806390 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.807145 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:51:59.807345 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:51:59.807422 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:51:59.807597 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:51:59.807703 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:51:59.807799 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.807824 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.807855 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:51:59.809753 4757 raft_consensus.cc:1261] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 1 FOLLOWER]: Refusing update from remote peer 2293a0d8eec14d29b128e99902b17334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:59.809772 4624 raft_consensus.cc:1261] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 1 FOLLOWER]: Refusing update from remote peer 2293a0d8eec14d29b128e99902b17334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:51:59.810117 4960 consensus_queue.cc:1035] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce09e565dc37465293b88850d15f2670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:59.810249 4956 consensus_queue.cc:1035] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd81fdbd6b7f411baa86152e0f7c58e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:51:59.812471 4963 mvcc.cc:205] Tried to move back new op lower bound from 7334370999538061312 to 7334370999451217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334370999538061312}]
I20260928 17:51:59.812850 4965 mvcc.cc:205] Tried to move back new op lower bound from 7334370999538061312 to 7334370999451217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334370999538061312}]
I20260928 17:51:59.813190 4964 mvcc.cc:205] Tried to move back new op lower bound from 7334370999538061312 to 7334370999451217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334370999538061312}]
I20260928 17:51:59.814692 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:59.814793 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:51:59.815557 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ec36ce2bc7d4de0943a54e09cc9d7df: Sending DeleteTablet for 3 replicas of tablet 14abf3e9224b42cd9652f208de0dee63
I20260928 17:51:59.815873 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 14abf3e9224b42cd9652f208de0dee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38162
I20260928 17:51:59.815929 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 14abf3e9224b42cd9652f208de0dee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59720
I20260928 17:51:59.816140 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 14abf3e9224b42cd9652f208de0dee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60076
I20260928 17:51:59.816138 4974 tablet_replica.cc:330] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: stopping tablet replica
I20260928 17:51:59.816303 4974 raft_consensus.cc:2229] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:59.816346 4975 tablet_replica.cc:330] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670: stopping tablet replica
I20260928 17:51:59.816310 4973 tablet_replica.cc:330] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: stopping tablet replica
I20260928 17:51:59.816416 4974 raft_consensus.cc:2258] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:59.816425 4975 raft_consensus.cc:2229] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:51:59.816497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4542
I20260928 17:51:59.816516 4975 raft_consensus.cc:2258] T 14abf3e9224b42cd9652f208de0dee63 P ce09e565dc37465293b88850d15f2670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:59.816550 4973 raft_consensus.cc:2229] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:51:59.816650 4973 raft_consensus.cc:2258] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:51:59.817447 4973 ts_tablet_manager.cc:1905] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:59.817736 4974 ts_tablet_manager.cc:1905] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:51:59.819156 4974 ts_tablet_manager.cc:1918] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:51:59.819240 4974 log.cc:1196] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/14abf3e9224b42cd9652f208de0dee63
I20260928 17:51:59.819430 4973 ts_tablet_manager.cc:1918] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:51:59.819504 4973 log.cc:1196] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/14abf3e9224b42cd9652f208de0dee63
I20260928 17:51:59.819528 4974 ts_tablet_manager.cc:1939] T 14abf3e9224b42cd9652f208de0dee63 P fd81fdbd6b7f411baa86152e0f7c58e0: Deleting consensus metadata
I20260928 17:51:59.819768 4973 ts_tablet_manager.cc:1939] T 14abf3e9224b42cd9652f208de0dee63 P 2293a0d8eec14d29b128e99902b17334: Deleting consensus metadata
I20260928 17:51:59.819906 4466 catalog_manager.cc:4841] TS fd81fdbd6b7f411baa86152e0f7c58e0 (127.0.98.2:40405): tablet 14abf3e9224b42cd9652f208de0dee63 (table test-table [id=3f5a35d1d26f433bbdb0331d55c1bb2d]) successfully deleted
I20260928 17:51:59.820549 4466 catalog_manager.cc:4841] TS 2293a0d8eec14d29b128e99902b17334 (127.0.98.3:45625): tablet 14abf3e9224b42cd9652f208de0dee63 (table test-table [id=3f5a35d1d26f433bbdb0331d55c1bb2d]) successfully deleted
W20260928 17:51:59.822357 4467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46723 (error 108)
I20260928 17:51:59.823511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4675
I20260928 17:51:59.828538 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.828397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46723 (local address 127.0.0.1:60080)
0928 17:51:59.828456 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:51:59.828505 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 17:51:59.828707 4467 catalog_manager.cc:4584] TS ce09e565dc37465293b88850d15f2670 (127.0.98.1:46723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14abf3e9224b42cd9652f208de0dee63: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46723: connect: Connection refused (error 111)
I20260928 17:51:59.830727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4808
I20260928 17:51:59.835486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4449
2026-09-28T17:51:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1948 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T17:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:51:59Z Disabled control of system clock
I20260928 17:51:59.863577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36167
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:51:59.943342 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:51:59.943614 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:51:59.943650 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:51:59.944864 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:51:59.944942 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:51:59.944963 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:51:59.944984 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:51:59.945014 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:51:59.946391 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:51:59.946678 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:51:59.946987 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:51:59.952075 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.952137 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:51:59.952148 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:51:59.952318 4980 server_base.cc:1034] running on GCE node
I20260928 17:51:59.952611 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:51:59.952951 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:51:59.954139 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790617919954124 us; error 27 us; skew 500 ppm
I20260928 17:51:59.955484 4980 webserver.cc:460] Webserver started at http://127.0.98.62:40763/ using document root <none> and password file <none>
I20260928 17:51:59.955768 4980 fs_manager.cc:359] Metadata directory not provided
I20260928 17:51:59.955840 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:51:59.955997 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:51:59.957181 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "05f0ada7ab8d43c7a4fae174912d7f83"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "8a3fca93ba241b5398cf22da31a2798f"
server_key_iv: "1ecdd0aa5dcf261c4c214646b617c329"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.957578 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "05f0ada7ab8d43c7a4fae174912d7f83"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "8a3fca93ba241b5398cf22da31a2798f"
server_key_iv: "1ecdd0aa5dcf261c4c214646b617c329"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.958907 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:51:59.959630 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:51:59.959801 4980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:51:59.959895 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "05f0ada7ab8d43c7a4fae174912d7f83"
format_stamp: "Formatted at 2026-09-28 17:51:59 on dist-test-slave-zqlt"
server_key: "8a3fca93ba241b5398cf22da31a2798f"
server_key_iv: "1ecdd0aa5dcf261c4c214646b617c329"
server_key_version: "encryptionkey@0"
I20260928 17:51:59.959978 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:51:59.963546 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:51:59.963827 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:51:59.963992 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:51:59.967907 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36167
I20260928 17:51:59.967935 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36167 every 8 connection(s)
I20260928 17:51:59.968335 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:51:59.968966 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:51:59.971653 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: Bootstrap starting.
I20260928 17:51:59.972290 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: Neither blocks nor log segments found. Creating new log.
I20260928 17:51:59.972584 5047 log.cc:824] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: Log is configured to *not* fsync() on all Append() calls
I20260928 17:51:59.973225 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: No bootstrap required, opened a new log
I20260928 17:51:59.974757 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } }
I20260928 17:51:59.974895 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:51:59.974916 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f0ada7ab8d43c7a4fae174912d7f83, State: Initialized, Role: FOLLOWER
I20260928 17:51:59.975081 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } }
I20260928 17:51:59.975140 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:51:59.975162 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:51:59.975219 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:51:59.975739 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } }
I20260928 17:51:59.975847 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f0ada7ab8d43c7a4fae174912d7f83; no voters:
I20260928 17:51:59.976022 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:51:59.976073 5050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:51:59.976300 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:51:59.976323 5050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [term 1 LEADER]: Becoming Leader. State: Replica: 05f0ada7ab8d43c7a4fae174912d7f83, State: Running, Role: LEADER
I20260928 17:51:59.976402 5050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } }
I20260928 17:51:59.976984 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } } }
I20260928 17:51:59.977083 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:59.977399 5059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:51:59.977519 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f0ada7ab8d43c7a4fae174912d7f83. Latest consensus state: current_term: 1 leader_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f0ada7ab8d43c7a4fae174912d7f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36167 } } }
I20260928 17:51:59.977589 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83 [sys.catalog]: This master's current role is: LEADER
I20260928 17:51:59.977849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 4980
I20260928 17:51:59.977933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:51:59.978116 392 external_mini_cluster.cc:1428] Setting key a015e0b9900e3179b2e508f01b8853a5
I20260928 17:51:59.978297 5059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:51:59.980527 5059 catalog_manager.cc:1347] Generated new cluster ID: d0f32c2378c24ffa8179730522136e32
I20260928 17:51:59.980584 5059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:51:59.980568 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:51:59.978559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.0.98.62:36167)
0928 17:51:59.978786 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 17:51:59.978795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:51:59.978827 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:51:59.978951 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:51:59.978958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:51:59.979003 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:51:59.979146 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:51:59.980035 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:51:59.980038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:51:59.980038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:51:59.980041 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:51:59.980080 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:51:59.980130 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:51:59.980218 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:51:59.980300 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 17:51:59.980438 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260928 17:51:59.984479 5059 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:51:59.984645 5059 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:51:59.989375 5059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: Generated new TSK 0
I20260928 17:51:59.989530 5059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:51:59.994231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--builtin_ntp_servers=127.0.98.20:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:52:00.076375 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:00.076609 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:00.076645 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:00.078097 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:00.078138 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:00.078166 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:00.079560 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:00.079797 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:00.080137 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:00.084965 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.084939 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.084928 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.085142 5071 server_base.cc:1034] running on GCE node
I20260928 17:52:00.085325 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:00.085594 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:00.086743 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790617920086728 us; error 35 us; skew 500 ppm
I20260928 17:52:00.088040 5071 webserver.cc:460] Webserver started at http://127.0.98.1:42569/ using document root <none> and password file <none>
I20260928 17:52:00.088310 5071 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:00.088354 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:00.088487 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:00.089560 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "676d67d339994ca98ad309c3ed6929d9"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "38fbc590e2aa4ea6fc5838a22cdb48af"
server_key_iv: "9cf93549ea2751905363c51c3f59c96e"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.089918 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "676d67d339994ca98ad309c3ed6929d9"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "38fbc590e2aa4ea6fc5838a22cdb48af"
server_key_iv: "9cf93549ea2751905363c51c3f59c96e"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.091226 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:00.092001 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.092211 5071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:00.092307 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "676d67d339994ca98ad309c3ed6929d9"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "38fbc590e2aa4ea6fc5838a22cdb48af"
server_key_iv: "9cf93549ea2751905363c51c3f59c96e"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.092386 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:00.095619 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:00.095904 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:00.096046 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:00.096298 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:00.096637 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:00.096690 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.096729 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:00.096755 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.103853 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44537
I20260928 17:52:00.103880 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44537 every 8 connection(s)
I20260928 17:52:00.104274 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:00.107414 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.105068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52191 (local address 127.0.98.62:36167)
0928 17:52:00.105117 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.105119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.105367 (+ 248us) server_negotiation.cc:407] Connection header received
0928 17:52:00.105990 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.105991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.105997 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.106039 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.107192 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.107194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.107194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.107199 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.107214 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.107218 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.107267 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.107350 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.107363 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:52:00.107679 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.105065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36167 (local address 127.0.98.1:52191)
0928 17:52:00.105242 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.105285 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.105883 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.106109 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.106113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.106123 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.107047 (+ 924us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.107047 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.107075 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.107103 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.107280 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.107284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.107287 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.107344 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.107392 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260928 17:52:00.108222 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36167
I20260928 17:52:00.108304 5199 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:00.108407 5199 heartbeater.cc:511] Master 127.0.98.62:36167 requested a full tablet report, sending...
I20260928 17:52:00.108839 5011 ts_manager.cc:194] Registered new tserver with Master: 676d67d339994ca98ad309c3ed6929d9 (127.0.98.1:44537)
I20260928 17:52:00.109000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5071
I20260928 17:52:00.109100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:00.109266 392 external_mini_cluster.cc:1428] Setting key 12d1efbac880648cd672128806f16285
I20260928 17:52:00.110049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--builtin_ntp_servers=127.0.98.20:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:52:00.189531 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:00.189769 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:00.189815 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:00.191013 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:00.191075 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:00.191118 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:00.192516 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:00.192781 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:00.193074 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:00.198491 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.198529 5202 server_base.cc:1034] running on GCE node
W20260928 17:52:00.198531 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.198659 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.198863 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:00.199059 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:00.200158 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790617920200143 us; error 40 us; skew 500 ppm
I20260928 17:52:00.201344 5202 webserver.cc:460] Webserver started at http://127.0.98.2:39501/ using document root <none> and password file <none>
I20260928 17:52:00.201577 5202 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:00.201623 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:00.201788 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:00.202836 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac1f5bd9537341f1afefb263bf1e6431"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "6269718093b83c17a9071c30b851ccbf"
server_key_iv: "cafa10ec68b98141f25c2b3a8bbddc15"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.203194 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac1f5bd9537341f1afefb263bf1e6431"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "6269718093b83c17a9071c30b851ccbf"
server_key_iv: "cafa10ec68b98141f25c2b3a8bbddc15"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.204603 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:00.205287 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.205485 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:00.205555 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "ac1f5bd9537341f1afefb263bf1e6431"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "6269718093b83c17a9071c30b851ccbf"
server_key_iv: "cafa10ec68b98141f25c2b3a8bbddc15"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.205642 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:00.209071 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:00.209336 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:00.209472 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:00.209672 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:00.209959 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:00.209990 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.210011 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:00.210057 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.215910 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42567
I20260928 17:52:00.215946 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42567 every 8 connection(s)
I20260928 17:52:00.216353 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:00.219264 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.217086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51875 (local address 127.0.98.62:36167)
0928 17:52:00.217174 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.217177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.217380 (+ 203us) server_negotiation.cc:407] Connection header received
0928 17:52:00.217841 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.217842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.217847 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.217902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.219093 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.219094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.219095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.219097 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.219108 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.219111 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.219154 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.219201 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.219212 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 17:52:00.219480 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.217060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36167 (local address 127.0.98.2:51875)
0928 17:52:00.217257 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.217291 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.217754 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.217958 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.217961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.217969 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.218960 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.218960 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.218989 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.219019 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.219153 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.219155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.219158 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.219209 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.219257 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260928 17:52:00.219974 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36167
I20260928 17:52:00.220052 5330 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:00.220232 5330 heartbeater.cc:511] Master 127.0.98.62:36167 requested a full tablet report, sending...
I20260928 17:52:00.220609 5011 ts_manager.cc:194] Registered new tserver with Master: ac1f5bd9537341f1afefb263bf1e6431 (127.0.98.2:42567)
I20260928 17:52:00.224769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5202
I20260928 17:52:00.224869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:00.225006 392 external_mini_cluster.cc:1428] Setting key 48435baab992163d832d361a927be695
I20260928 17:52:00.225903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--builtin_ntp_servers=127.0.98.20:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 17:52:00.306535 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:00.306731 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:00.306753 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:00.307904 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:00.307943 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:00.307968 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:00.309410 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:00.309645 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:00.309930 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:00.314755 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.314769 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.314775 5333 server_base.cc:1034] running on GCE node
W20260928 17:52:00.314759 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.315151 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:00.315398 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:00.316574 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790617920316562 us; error 25 us; skew 500 ppm
I20260928 17:52:00.317806 5333 webserver.cc:460] Webserver started at http://127.0.98.3:39119/ using document root <none> and password file <none>
I20260928 17:52:00.318063 5333 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:00.318132 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:00.318262 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:00.319268 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9fa9a1335344b7e9c668a1081dd9a54"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "8abf9dca57855c06b6a3d17dc72c380c"
server_key_iv: "b1eb0c540f1ecf44c4a1719ff9d22033"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.319633 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9fa9a1335344b7e9c668a1081dd9a54"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "8abf9dca57855c06b6a3d17dc72c380c"
server_key_iv: "b1eb0c540f1ecf44c4a1719ff9d22033"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.321030 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:00.321743 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.321920 5333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:00.322014 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "d9fa9a1335344b7e9c668a1081dd9a54"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "8abf9dca57855c06b6a3d17dc72c380c"
server_key_iv: "b1eb0c540f1ecf44c4a1719ff9d22033"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.322096 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:00.325400 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:00.325662 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:00.325801 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:00.326021 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:00.326313 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:00.326364 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.326401 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:00.326428 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.332023 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39655
I20260928 17:52:00.332118 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39655 every 8 connection(s)
I20260928 17:52:00.332442 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:00.335373 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.333166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38711 (local address 127.0.98.62:36167)
0928 17:52:00.333223 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.333225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.333443 (+ 218us) server_negotiation.cc:407] Connection header received
0928 17:52:00.333955 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.333957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.333964 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.334011 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.335113 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.335115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.335115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.335120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.335135 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.335140 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.335194 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.335287 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.335302 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:52:00.335553 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.333160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36167 (local address 127.0.98.3:38711)
0928 17:52:00.333330 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.333361 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.333837 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.334095 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.334098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.334106 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.334964 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.334964 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.334992 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.335019 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.335189 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.335192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.335195 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.335291 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.335352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 17:52:00.335968 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36167
I20260928 17:52:00.336052 5461 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:00.336220 5461 heartbeater.cc:511] Master 127.0.98.62:36167 requested a full tablet report, sending...
I20260928 17:52:00.336537 5011 ts_manager.cc:194] Registered new tserver with Master: d9fa9a1335344b7e9c668a1081dd9a54 (127.0.98.3:39655)
I20260928 17:52:00.340312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5333
I20260928 17:52:00.340415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:00.340590 392 external_mini_cluster.cc:1428] Setting key a095b7e07daf762c9c89fb57ed061226
I20260928 17:52:00.341534 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:52:00.343935 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.342863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.0.98.62:36167)
0928 17:52:00.342932 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.342934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.343002 (+ 68us) server_negotiation.cc:407] Connection header received
0928 17:52:00.343039 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.343040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.343044 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.343095 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.343706 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.343707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.343707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.343710 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.343720 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.343724 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.343762 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.343810 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.343821 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:52:00.345357 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:00.346865 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:00.350718 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.349005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39655 (local address 127.0.0.1:56928)
0928 17:52:00.349492 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.349502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.349558 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.349926 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.349927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.349929 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.350302 (+ 373us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.350303 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.350319 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.350344 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.350547 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.350553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.350556 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.350630 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.350644 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":93,"threads_started":1}
I20260928 17:52:00.350921 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.349007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44537 (local address 127.0.0.1:55902)
0928 17:52:00.349646 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.349656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.349696 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.350163 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.350164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.350166 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.350543 (+ 377us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.350543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.350551 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.350564 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.350761 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.350761 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.350762 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.350827 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.350841 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":84,"threads_started":1}
I20260928 17:52:00.351104 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.349352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.0.98.3:39655)
0928 17:52:00.349477 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.349482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.349582 (+ 100us) server_negotiation.cc:407] Connection header received
0928 17:52:00.349612 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.349622 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.349631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.350093 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.350417 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.350422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.350423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.350426 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.350457 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.350472 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.350555 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.350992 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.351013 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260928 17:52:00.351119 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.349240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.98.1:44537)
0928 17:52:00.349397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.349401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.349753 (+ 352us) server_negotiation.cc:407] Connection header received
0928 17:52:00.349785 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.349787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.349798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.349865 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.350633 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.350640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.350640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.350644 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.350675 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.350698 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.350759 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.351017 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.351037 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260928 17:52:00.351729 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.348988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42567 (local address 127.0.0.1:60790)
0928 17:52:00.349269 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.349313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.349379 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.349557 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.349563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.349570 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.350543 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.350543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.350551 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.350564 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.351595 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.351596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.351597 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.351654 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.351666 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":71,"threads_started":1}
I20260928 17:52:00.351811 5133 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb63174c04c398b4683d10b843e9e (DEFAULT_TABLE table=test-table [id=fc07663c426644e7a16289762364a369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:00.351800 5395 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb63174c04c398b4683d10b843e9e (DEFAULT_TABLE table=test-table [id=fc07663c426644e7a16289762364a369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:00.352123 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.349116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.0.98.2:42567)
0928 17:52:00.349300 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.349306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.349440 (+ 134us) server_negotiation.cc:407] Connection header received
0928 17:52:00.349467 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.349469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.349488 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.349565 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.350799 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.350808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.350808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.350813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.350850 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.350877 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.350933 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.351923 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.351941 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260928 17:52:00.352186 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb63174c04c398b4683d10b843e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:00.352195 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb63174c04c398b4683d10b843e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:00.355151 5476 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Bootstrap starting.
I20260928 17:52:00.355293 5475 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: Bootstrap starting.
I20260928 17:52:00.355430 5264 tablet_service.cc:1467] Processing CreateTablet for tablet d68cb63174c04c398b4683d10b843e9e (DEFAULT_TABLE table=test-table [id=fc07663c426644e7a16289762364a369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:00.355779 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68cb63174c04c398b4683d10b843e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:00.355916 5476 tablet_bootstrap.cc:654] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:00.355996 5475 tablet_bootstrap.cc:654] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:00.356266 5476 log.cc:824] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:00.356302 5475 log.cc:824] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:00.357061 5476 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: No bootstrap required, opened a new log
I20260928 17:52:00.357144 5476 ts_tablet_manager.cc:1397] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:52:00.358686 5479 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Bootstrap starting.
I20260928 17:52:00.359169 5476 raft_consensus.cc:348] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.359351 5476 raft_consensus.cc:374] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:00.359382 5479 tablet_bootstrap.cc:654] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:00.359395 5476 raft_consensus.cc:729] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9fa9a1335344b7e9c668a1081dd9a54, State: Initialized, Role: FOLLOWER
I20260928 17:52:00.359534 5476 consensus_queue.cc:260] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.359696 5479 log.cc:824] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:00.359860 5461 heartbeater.cc:503] Master 127.0.98.62:36167 was elected leader, sending a full tablet report...
I20260928 17:52:00.360464 5475 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: No bootstrap required, opened a new log
I20260928 17:52:00.360538 5475 ts_tablet_manager.cc:1397] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 17:52:00.361017 5479 tablet_bootstrap.cc:492] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: No bootstrap required, opened a new log
I20260928 17:52:00.361092 5479 ts_tablet_manager.cc:1397] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:52:00.363298 5475 raft_consensus.cc:348] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.363540 5475 raft_consensus.cc:374] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:00.363581 5475 raft_consensus.cc:729] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676d67d339994ca98ad309c3ed6929d9, State: Initialized, Role: FOLLOWER
I20260928 17:52:00.363708 5475 consensus_queue.cc:260] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.363761 5479 raft_consensus.cc:348] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.363921 5479 raft_consensus.cc:374] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:00.363962 5479 raft_consensus.cc:729] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1f5bd9537341f1afefb263bf1e6431, State: Initialized, Role: FOLLOWER
I20260928 17:52:00.364008 5475 ts_tablet_manager.cc:1428] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 17:52:00.364054 5199 heartbeater.cc:503] Master 127.0.98.62:36167 was elected leader, sending a full tablet report...
I20260928 17:52:00.364137 5476 ts_tablet_manager.cc:1428] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260928 17:52:00.364117 5479 consensus_queue.cc:260] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.364553 5330 heartbeater.cc:503] Master 127.0.98.62:36167 was elected leader, sending a full tablet report...
I20260928 17:52:00.364619 5479 ts_tablet_manager.cc:1428] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 17:52:00.437664 5483 raft_consensus.cc:482] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:00.437847 5483 raft_consensus.cc:504] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.438308 5483 leader_election.cc:288] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9fa9a1335344b7e9c668a1081dd9a54 (127.0.98.3:39655), 676d67d339994ca98ad309c3ed6929d9 (127.0.98.1:44537)
I20260928 17:52:00.440140 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.438645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51989 (local address 127.0.98.3:39655)
0928 17:52:00.438721 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.438723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.438855 (+ 132us) server_negotiation.cc:407] Connection header received
0928 17:52:00.438953 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.438954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.438960 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.438992 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.439816 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.439817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.439818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.439823 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.439842 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.439847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.439912 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.440015 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.440032 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 17:52:00.440254 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.438768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59475 (local address 127.0.98.1:44537)
0928 17:52:00.438820 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.438822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.438831 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:00.438859 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.438860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.438866 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.438934 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.439917 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.439919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.439919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.439924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.439956 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.439960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.440013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.440130 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.440145 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:52:00.440284 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.438515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39655 (local address 127.0.98.2:51989)
0928 17:52:00.438793 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.438804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.438878 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.439064 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.439065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.439067 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.439635 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.439636 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.439646 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.439689 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.439943 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.439944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.439945 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.440174 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.440189 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":74,"thread_start_us":112,"threads_started":1}
I20260928 17:52:00.440442 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.438515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44537 (local address 127.0.98.2:59475)
0928 17:52:00.438584 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.438596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.438662 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.438951 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.438952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.438954 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:00.439822 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:00.439822 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:00.439827 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:00.439852 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:00.440009 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:00.440009 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:00.440010 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:52:00.440372 (+ 362us) client_negotiation.cc:240] Negotiation successful
0928 17:52:00.440388 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 17:52:00.440788 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb63174c04c398b4683d10b843e9e" candidate_uuid: "ac1f5bd9537341f1afefb263bf1e6431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" is_pre_election: true
I20260928 17:52:00.440979 5415 raft_consensus.cc:2454] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1f5bd9537341f1afefb263bf1e6431 in term 0.
I20260928 17:52:00.441222 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb63174c04c398b4683d10b843e9e" candidate_uuid: "ac1f5bd9537341f1afefb263bf1e6431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676d67d339994ca98ad309c3ed6929d9" is_pre_election: true
I20260928 17:52:00.441295 5218 leader_election.cc:302] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac1f5bd9537341f1afefb263bf1e6431, d9fa9a1335344b7e9c668a1081dd9a54; no voters:
I20260928 17:52:00.441373 5153 raft_consensus.cc:2454] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1f5bd9537341f1afefb263bf1e6431 in term 0.
I20260928 17:52:00.441591 5483 raft_consensus.cc:2781] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:00.441668 5483 raft_consensus.cc:482] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:00.441696 5483 raft_consensus.cc:3037] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:00.442446 5483 raft_consensus.cc:504] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.442596 5483 leader_election.cc:288] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [CANDIDATE]: Term 1 election: Requested vote from peers d9fa9a1335344b7e9c668a1081dd9a54 (127.0.98.3:39655), 676d67d339994ca98ad309c3ed6929d9 (127.0.98.1:44537)
I20260928 17:52:00.442723 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb63174c04c398b4683d10b843e9e" candidate_uuid: "ac1f5bd9537341f1afefb263bf1e6431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9fa9a1335344b7e9c668a1081dd9a54"
I20260928 17:52:00.442739 5153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68cb63174c04c398b4683d10b843e9e" candidate_uuid: "ac1f5bd9537341f1afefb263bf1e6431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676d67d339994ca98ad309c3ed6929d9"
I20260928 17:52:00.442818 5153 raft_consensus.cc:3037] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:00.442831 5415 raft_consensus.cc:3037] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:00.443523 5415 raft_consensus.cc:2454] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1f5bd9537341f1afefb263bf1e6431 in term 1.
I20260928 17:52:00.443617 5153 raft_consensus.cc:2454] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1f5bd9537341f1afefb263bf1e6431 in term 1.
I20260928 17:52:00.443675 5218 leader_election.cc:302] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac1f5bd9537341f1afefb263bf1e6431, d9fa9a1335344b7e9c668a1081dd9a54; no voters:
I20260928 17:52:00.443758 5483 raft_consensus.cc:2781] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:00.443915 5483 raft_consensus.cc:686] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 1 LEADER]: Becoming Leader. State: Replica: ac1f5bd9537341f1afefb263bf1e6431, State: Running, Role: LEADER
I20260928 17:52:00.444010 5483 consensus_queue.cc:237] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } }
I20260928 17:52:00.444904 5009 catalog_manager.cc:5495] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac1f5bd9537341f1afefb263bf1e6431 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ac1f5bd9537341f1afefb263bf1e6431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1f5bd9537341f1afefb263bf1e6431" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 } health_report { overall_health: UNKNOWN } } }
W20260928 17:52:00.466878 5331 tablet.cc:2369] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:00.467600 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.466469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.0.98.2:42567)
0928 17:52:00.466534 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.466536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.466547 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:00.466613 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.466614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.466629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.466705 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.467357 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.467359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.467359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.467370 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.467386 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.467391 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.467449 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.467510 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.467525 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:52:00.469651 5415 raft_consensus.cc:1261] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 1 FOLLOWER]: Refusing update from remote peer ac1f5bd9537341f1afefb263bf1e6431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:00.469717 5153 raft_consensus.cc:1261] T d68cb63174c04c398b4683d10b843e9e P 676d67d339994ca98ad309c3ed6929d9 [term 1 FOLLOWER]: Refusing update from remote peer ac1f5bd9537341f1afefb263bf1e6431: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:00.469995 5485 consensus_queue.cc:1035] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9fa9a1335344b7e9c668a1081dd9a54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:00.470120 5483 consensus_queue.cc:1035] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676d67d339994ca98ad309c3ed6929d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:00.473316 5489 mvcc.cc:205] Tried to move back new op lower bound from 7334371002240860160 to 7334371002139496448. Current Snapshot: MvccSnapshot[applied={T|T < 7334371002240860160}]
I20260928 17:52:00.474357 5490 mvcc.cc:205] Tried to move back new op lower bound from 7334371002240860160 to 7334371002139496448. Current Snapshot: MvccSnapshot[applied={T|T < 7334371002240860160 or (T in {7334371002240860160})}]
I20260928 17:52:00.474638 5011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:00.474740 5011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:00.475495 5011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f0ada7ab8d43c7a4fae174912d7f83: Sending DeleteTablet for 3 replicas of tablet d68cb63174c04c398b4683d10b843e9e
I20260928 17:52:00.475811 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb63174c04c398b4683d10b843e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:00 UTC) from {username='slave'} at 127.0.0.1:60790
I20260928 17:52:00.475880 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb63174c04c398b4683d10b843e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:00 UTC) from {username='slave'} at 127.0.0.1:55902
I20260928 17:52:00.475888 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet d68cb63174c04c398b4683d10b843e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:00 UTC) from {username='slave'} at 127.0.0.1:56928
I20260928 17:52:00.476030 5497 tablet_replica.cc:330] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: stopping tablet replica
I20260928 17:52:00.476132 5499 tablet_replica.cc:330] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: stopping tablet replica
I20260928 17:52:00.476140 5497 raft_consensus.cc:2229] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:00.476219 5499 raft_consensus.cc:2229] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:00.476276 5497 raft_consensus.cc:2258] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:00.476322 5499 raft_consensus.cc:2258] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:00.476388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5071
I20260928 17:52:00.476785 5497 ts_tablet_manager.cc:1905] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:00.478641 5499 ts_tablet_manager.cc:1905] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:00.479024 5497 ts_tablet_manager.cc:1918] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:00.479162 5497 log.cc:1196] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/d68cb63174c04c398b4683d10b843e9e
I20260928 17:52:00.479444 5497 ts_tablet_manager.cc:1939] T d68cb63174c04c398b4683d10b843e9e P ac1f5bd9537341f1afefb263bf1e6431: Deleting consensus metadata
I20260928 17:52:00.479876 4998 catalog_manager.cc:4841] TS ac1f5bd9537341f1afefb263bf1e6431 (127.0.98.2:42567): tablet d68cb63174c04c398b4683d10b843e9e (table test-table [id=fc07663c426644e7a16289762364a369]) successfully deleted
I20260928 17:52:00.480686 5499 ts_tablet_manager.cc:1918] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:00.480765 5499 log.cc:1196] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/d68cb63174c04c398b4683d10b843e9e
I20260928 17:52:00.481053 5499 ts_tablet_manager.cc:1939] T d68cb63174c04c398b4683d10b843e9e P d9fa9a1335344b7e9c668a1081dd9a54: Deleting consensus metadata
I20260928 17:52:00.481504 4996 catalog_manager.cc:4841] TS d9fa9a1335344b7e9c668a1081dd9a54 (127.0.98.3:39655): tablet d68cb63174c04c398b4683d10b843e9e (table test-table [id=fc07663c426644e7a16289762364a369]) successfully deleted
W20260928 17:52:00.483506 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44537 (error 108)
I20260928 17:52:00.483724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5202
I20260928 17:52:00.484725 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.483923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44537 (local address 127.0.0.1:55916)
0928 17:52:00.484653 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.484696 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":705}
W20260928 17:52:00.484861 4995 catalog_manager.cc:4584] TS 676d67d339994ca98ad309c3ed6929d9 (127.0.98.1:44537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d68cb63174c04c398b4683d10b843e9e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44537: connect: Connection refused (error 111)
I20260928 17:52:00.489630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5333
I20260928 17:52:00.494499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 4980
2026-09-28T17:52:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (657 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 17:52:00.513561 392 mini_oidc.cc:152] Starting JWKS server
I20260928 17:52:00.514578 392 webserver.cc:460] Webserver started at https://127.0.0.1:38059/ using document root <none> and password file <none>
I20260928 17:52:00.514694 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:52:00.514882 392 webserver.cc:460] Webserver started at http://127.0.0.1:34473/ using document root <none> and password file <none>
2026-09-28T17:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:00Z Disabled control of system clock
I20260928 17:52:00.522260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38857
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/testchainca.cert with env {}
W20260928 17:52:00.605235 5506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:00.605454 5506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:00.605477 5506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:00.606619 5506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:00.606655 5506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:00.606668 5506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:00.606681 5506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:00.606693 5506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:52:00.606705 5506 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:52:00.606724 5506 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38059/jwks.json
I20260928 17:52:00.608285 5506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:00.608486 5506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:00.608646 5506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:00.613376 5514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.613441 5511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.613369 5512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.613624 5506 server_base.cc:1034] running on GCE node
I20260928 17:52:00.613966 5506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:00.614277 5506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:00.615469 5506 hybrid_clock.cc:648] HybridClock initialized: now 1790617920615451 us; error 29 us; skew 500 ppm
I20260928 17:52:00.616710 5506 webserver.cc:460] Webserver started at http://127.0.98.62:33727/ using document root <none> and password file <none>
I20260928 17:52:00.616955 5506 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:00.617025 5506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:00.617184 5506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:00.618287 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "1515d0ba18964945ab9614d6a6a300a0"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "ef66d29ffc26c803d37e3d8bdfc7856d"
server_key_iv: "79e6a03c3050b04de5492adebf716888"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.618677 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "1515d0ba18964945ab9614d6a6a300a0"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "ef66d29ffc26c803d37e3d8bdfc7856d"
server_key_iv: "79e6a03c3050b04de5492adebf716888"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.619974 5506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:00.620863 5520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.621057 5506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:00.621158 5506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "1515d0ba18964945ab9614d6a6a300a0"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "ef66d29ffc26c803d37e3d8bdfc7856d"
server_key_iv: "79e6a03c3050b04de5492adebf716888"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.621250 5506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:00.636930 5506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:00.637282 5506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:00.637455 5506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:00.641511 5506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38857
I20260928 17:52:00.641578 5572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38857 every 8 connection(s)
I20260928 17:52:00.641950 5506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:00.642933 5573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:00.645403 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0: Bootstrap starting.
I20260928 17:52:00.646067 5573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:00.646391 5573 log.cc:824] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:00.646401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5506
I20260928 17:52:00.646526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:00.646710 392 external_mini_cluster.cc:1428] Setting key c54cf8b5d60ce229f95417a1f5edaf47
I20260928 17:52:00.647256 5573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0: No bootstrap required, opened a new log
I20260928 17:52:00.650158 5573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260928 17:52:00.650348 5573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:00.650380 5573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1515d0ba18964945ab9614d6a6a300a0, State: Initialized, Role: FOLLOWER
I20260928 17:52:00.650501 5573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260928 17:52:00.650573 5573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:00.650609 5573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:00.650661 5573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:00.651245 5573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260928 17:52:00.651357 5573 leader_election.cc:302] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1515d0ba18964945ab9614d6a6a300a0; no voters:
I20260928 17:52:00.651512 5573 leader_election.cc:288] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:00.651566 5578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:00.651731 5578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1515d0ba18964945ab9614d6a6a300a0, State: Running, Role: LEADER
I20260928 17:52:00.651801 5573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:00.651839 5578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } }
I20260928 17:52:00.652192 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1515d0ba18964945ab9614d6a6a300a0. Latest consensus state: current_term: 1 leader_uuid: "1515d0ba18964945ab9614d6a6a300a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } } }
I20260928 17:52:00.652272 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:00.652479 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1515d0ba18964945ab9614d6a6a300a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1515d0ba18964945ab9614d6a6a300a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38857 } } }
I20260928 17:52:00.652547 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:00.653151 5584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:00.653735 5584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:00.654172 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.647266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.62:38857)
0928 17:52:00.647573 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.647584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.647615 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:52:00.648650 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.648662 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.648752 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.648926 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.649150 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.650559 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.651376 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.652022 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.652200 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.652994 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.652998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.652999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.653249 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.653395 (+ 146us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.653410 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.653838 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.653874 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.653959 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":166,"threads_started":1}
I20260928 17:52:00.655148 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:52:00.655444 5584 catalog_manager.cc:1347] Generated new cluster ID: 4719f1b59ded4db5914f541ac380ff51
I20260928 17:52:00.655501 5584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:00.664499 5584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:00.665215 5584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:00.669715 5584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1515d0ba18964945ab9614d6a6a300a0: Generated new TSK 0
I20260928 17:52:00.669905 5584 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 17:52:00.694367 5603 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 17:52:00.694497 5603 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 17:52:00.695153 5602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:00.678931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38857 (local address 127.0.0.1:43408)
0928 17:52:00.679141 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.679153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.679218 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.679428 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.679428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.679430 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:52:00.679489 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:00.679491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.680447 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.680448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:00.681036 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:00.681038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.681518 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.681520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:00.681567 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.681587 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:52:00.694690 (+ 13103us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 17:52:00.694805 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 17:52:00.695530 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.679181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.98.62:38857)
0928 17:52:00.679250 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.679252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.679265 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:52:00.679307 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.679308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.679323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.679407 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:52:00.679556 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.680343 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.681124 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.681434 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.681515 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.681699 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:52:00.694492 (+ 12793us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 17:52:00.695451 (+ 959us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 17:52:00.695542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5506
2026-09-28T17:52:00Z chronyd exiting
I20260928 17:52:00.731204 392 test_util.cc:175] -----------------------------------------------
I20260928 17:52:00.731283 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790617898245922-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 17:52:00.732065 392 mini_oidc.cc:152] Starting JWKS server
I20260928 17:52:00.732820 392 webserver.cc:460] Webserver started at https://127.0.0.1:34137/ using document root <none> and password file <none>
I20260928 17:52:00.732858 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:52:00.733031 392 webserver.cc:460] Webserver started at http://127.0.0.1:37709/ using document root <none> and password file <none>
2026-09-28T17:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:00Z Disabled control of system clock
I20260928 17:52:00.740689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38641
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38641
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/testchainca.cert with env {}
W20260928 17:52:00.821470 5609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:00.821692 5609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:00.821715 5609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:00.822847 5609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:00.822883 5609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:00.822896 5609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:00.822909 5609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:00.822922 5609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:52:00.822934 5609 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:52:00.822945 5609 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34137/jwks.json
I20260928 17:52:00.824505 5609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:00.824709 5609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:00.824878 5609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:00.829895 5617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.829977 5615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:00.829897 5614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:00.830245 5609 server_base.cc:1034] running on GCE node
I20260928 17:52:00.830509 5609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:00.830862 5609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:00.832036 5609 hybrid_clock.cc:648] HybridClock initialized: now 1790617920832013 us; error 43 us; skew 500 ppm
I20260928 17:52:00.833325 5609 webserver.cc:460] Webserver started at http://127.0.98.62:44995/ using document root <none> and password file <none>
I20260928 17:52:00.833580 5609 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:00.833649 5609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:00.833807 5609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:00.834957 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "f9a271987a26400e93855ba25dcee0ef"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "35c828f750388d894e772c6db5470e5d"
server_key_iv: "93e62acc74a58a921422385bbc86084d"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.835348 5609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9a271987a26400e93855ba25dcee0ef"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "35c828f750388d894e772c6db5470e5d"
server_key_iv: "93e62acc74a58a921422385bbc86084d"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.836789 5609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:00.837543 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:00.837715 5609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:00.837826 5609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "f9a271987a26400e93855ba25dcee0ef"
format_stamp: "Formatted at 2026-09-28 17:52:00 on dist-test-slave-zqlt"
server_key: "35c828f750388d894e772c6db5470e5d"
server_key_iv: "93e62acc74a58a921422385bbc86084d"
server_key_version: "encryptionkey@0"
I20260928 17:52:00.837903 5609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:00.865976 5609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:00.866343 5609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:00.866509 5609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:00.870666 5609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38641
I20260928 17:52:00.870702 5675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38641 every 8 connection(s)
I20260928 17:52:00.871069 5609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:00.872223 5676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:00.874702 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: Bootstrap starting.
I20260928 17:52:00.875115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5609
I20260928 17:52:00.875196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:00.875362 392 external_mini_cluster.cc:1428] Setting key 1fe202dd7a12a7a3645d06479f6d2477
I20260928 17:52:00.875427 5676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:00.875799 5676 log.cc:824] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:00.876616 5676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: No bootstrap required, opened a new log
I20260928 17:52:00.878450 5676 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } }
I20260928 17:52:00.878624 5676 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:00.878676 5676 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a271987a26400e93855ba25dcee0ef, State: Initialized, Role: FOLLOWER
I20260928 17:52:00.878847 5676 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } }
I20260928 17:52:00.878937 5676 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:00.878986 5676 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:00.879038 5676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:00.879609 5676 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } }
I20260928 17:52:00.879736 5676 leader_election.cc:302] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a271987a26400e93855ba25dcee0ef; no voters:
I20260928 17:52:00.879926 5676 leader_election.cc:288] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:00.880064 5681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:00.880231 5676 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:00.880264 5681 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [term 1 LEADER]: Becoming Leader. State: Replica: f9a271987a26400e93855ba25dcee0ef, State: Running, Role: LEADER
I20260928 17:52:00.880358 5681 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } }
I20260928 17:52:00.880710 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a271987a26400e93855ba25dcee0ef. Latest consensus state: current_term: 1 leader_uuid: "f9a271987a26400e93855ba25dcee0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } } }
I20260928 17:52:00.880779 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:00.880908 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a271987a26400e93855ba25dcee0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a271987a26400e93855ba25dcee0ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38641 } } }
I20260928 17:52:00.881004 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.875916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.0.98.62:38641)
0928 17:52:00.876158 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.876166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.876199 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:52:00.876623 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.876631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.876686 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.876884 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:00.877094 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.877963 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.878768 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.879314 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.879450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.880217 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:00.880222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:00.880223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:00.880443 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:00.880484 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:00.880498 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:00.880538 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:00.880681 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 17:52:00.880763 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260928 17:52:00.881003 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:00.882161 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 17:52:00.882256 5696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:52:00.882309 5696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:52:00.882359 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:00.882551 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:00.884181 5688 catalog_manager.cc:1347] Generated new cluster ID: 0c1ad598177e433ebc12eb7020057f73
I20260928 17:52:00.884232 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:00.905051 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:00.905563 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:00.911628 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a271987a26400e93855ba25dcee0ef: Generated new TSK 0
I20260928 17:52:00.911820 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 17:52:00.943038 5706 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 17:52:00.943133 5706 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 17:52:00.943409 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:00.921633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.0.98.62:38641)
0928 17:52:00.921703 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:00.921706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:00.921848 (+ 142us) server_negotiation.cc:407] Connection header received
0928 17:52:00.921889 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:00.921890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:00.921901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:00.921966 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:52:00.922115 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.922599 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.923754 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.923942 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.923991 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.924208 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:52:00.943177 (+ 18969us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 17:52:00.943354 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 17:52:00.943482 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 17:52:00.943526 5705 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:00.921556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38641 (local address 127.0.0.1:40834)
0928 17:52:00.921741 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:00.921753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:00.921833 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:00.921986 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:00.921988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:00.921990 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:52:00.922050 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:00.922052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.922694 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.922695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:00.923670 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:00.923673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:00.924014 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:00.924018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:00.924080 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:00.924109 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:52:00.943312 (+ 19203us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 17:52:00.943427 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
I20260928 17:52:00.943852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5609
2026-09-28T17:52:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (249 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 17:52:00.981564 392 mini_oidc.cc:152] Starting JWKS server
I20260928 17:52:00.982249 392 webserver.cc:460] Webserver started at https://127.0.0.1:43655/ using document root <none> and password file <none>
I20260928 17:52:00.982290 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 17:52:00.982474 392 webserver.cc:460] Webserver started at http://127.0.0.1:36211/ using document root <none> and password file <none>
2026-09-28T17:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:00Z Disabled control of system clock
I20260928 17:52:00.989913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44987
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44987
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 17:52:01.069856 5712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:01.070077 5712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:01.070099 5712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:01.071233 5712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:01.071267 5712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:01.071280 5712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:01.071292 5712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:01.071305 5712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:52:01.071316 5712 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:52:01.071327 5712 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43655/jwks.json
I20260928 17:52:01.072624 5712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:01.072871 5712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:01.073284 5712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:01.078403 5717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:01.078410 5718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:01.078619 5712 server_base.cc:1034] running on GCE node
W20260928 17:52:01.078423 5720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:01.078918 5712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:01.079157 5712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:01.080323 5712 hybrid_clock.cc:648] HybridClock initialized: now 1790617921080307 us; error 27 us; skew 500 ppm
I20260928 17:52:01.081550 5712 webserver.cc:460] Webserver started at http://127.0.98.62:42281/ using document root <none> and password file <none>
I20260928 17:52:01.081807 5712 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:01.081876 5712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:01.082022 5712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:01.083084 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "b3337618837d491b96b9589e9ca3adb4"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "bb89fff9d9a83d2ef045edd7534bac5e"
server_key_iv: "d7e01b1c3b37ee29adb88a17a8a2da3a"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.083459 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3337618837d491b96b9589e9ca3adb4"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "bb89fff9d9a83d2ef045edd7534bac5e"
server_key_iv: "d7e01b1c3b37ee29adb88a17a8a2da3a"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.084818 5712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:01.085532 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:01.085700 5712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:01.085798 5712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "b3337618837d491b96b9589e9ca3adb4"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "bb89fff9d9a83d2ef045edd7534bac5e"
server_key_iv: "d7e01b1c3b37ee29adb88a17a8a2da3a"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.085882 5712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:01.109856 5712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:01.110215 5712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:01.110386 5712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:01.114485 5712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44987
I20260928 17:52:01.114523 5778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44987 every 8 connection(s)
I20260928 17:52:01.114900 5712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:01.115613 5779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:01.118389 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4: Bootstrap starting.
I20260928 17:52:01.118996 5779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:01.119293 5779 log.cc:824] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:01.119974 5779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4: No bootstrap required, opened a new log
I20260928 17:52:01.121531 5779 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260928 17:52:01.121671 5779 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:01.121718 5779 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3337618837d491b96b9589e9ca3adb4, State: Initialized, Role: FOLLOWER
I20260928 17:52:01.121908 5779 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260928 17:52:01.121991 5779 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:01.122040 5779 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:01.122089 5779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:01.122598 5779 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260928 17:52:01.122709 5779 leader_election.cc:302] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3337618837d491b96b9589e9ca3adb4; no voters:
I20260928 17:52:01.122907 5779 leader_election.cc:288] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:01.122954 5782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:01.123123 5782 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [term 1 LEADER]: Becoming Leader. State: Replica: b3337618837d491b96b9589e9ca3adb4, State: Running, Role: LEADER
I20260928 17:52:01.123162 5779 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:01.123217 5782 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } }
I20260928 17:52:01.123615 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3337618837d491b96b9589e9ca3adb4. Latest consensus state: current_term: 1 leader_uuid: "b3337618837d491b96b9589e9ca3adb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } } }
I20260928 17:52:01.123693 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:01.123821 5783 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3337618837d491b96b9589e9ca3adb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3337618837d491b96b9589e9ca3adb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44987 } } }
I20260928 17:52:01.123900 5783 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:01.123966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5712
I20260928 17:52:01.124060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:01.124261 392 external_mini_cluster.cc:1428] Setting key 91a3d5d3f3821704da6fc7fd79618674
I20260928 17:52:01.125628 5791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:01.126123 5791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:01.127636 5791 catalog_manager.cc:1347] Generated new cluster ID: 462f499f95214873bee3722ac2e96dee
I20260928 17:52:01.127689 5791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:01.130265 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:01.124953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.62:44987)
0928 17:52:01.125145 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:01.125154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:01.125187 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:52:01.125287 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:01.125294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:01.125346 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:01.125488 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:01.125685 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.126983 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.128174 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.128769 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.128884 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.129585 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:01.129589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:01.129590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:01.129795 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:01.129834 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:01.129847 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:01.129908 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:01.130005 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:52:01.130075 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260928 17:52:01.131361 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:52:01.135478 5791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:01.136185 5791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:01.142963 5791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3337618837d491b96b9589e9ca3adb4: Generated new TSK 0
I20260928 17:52:01.143162 5791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:52:01.151922 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:01.148518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.0.98.62:44987)
0928 17:52:01.148584 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:01.148587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:01.148711 (+ 124us) server_negotiation.cc:407] Connection header received
0928 17:52:01.148799 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:01.148800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:01.148813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:01.148898 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:52:01.149043 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.149523 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.150348 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.150546 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.150631 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.150831 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:52:01.151718 (+ 887us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 17:52:01.151853 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 17:52:01.151976 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 17:52:01.153990 5808 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:01.148467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44987 (local address 127.0.0.1:35892)
0928 17:52:01.148637 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:01.148646 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:01.148729 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:01.148897 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:01.148897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:01.148899 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:52:01.148961 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:01.148964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.149623 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.149624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:01.150238 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:01.150240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.150631 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.150639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:01.150697 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.150702 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:52:01.152058 (+ 1356us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 17:52:01.153893 (+ 1835us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43655/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1610,"thread_start_us":100,"threads_started":1}
I20260928 17:52:01.154204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5712
E20260928 17:52:01.164512 5809 webserver.cc:533] Webserver: sslize failed:
E20260928 17:52:01.164610 5809 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T17:52:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T17:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:01Z Disabled control of system clock
I20260928 17:52:01.198952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40569
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40569
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/testchainca.cert with env {}
W20260928 17:52:01.280974 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:01.281235 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:01.281270 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:01.282469 5813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:01.282541 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:01.282565 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:01.282598 5813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:01.282624 5813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 17:52:01.282652 5813 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 17:52:01.282680 5813 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 17:52:01.284255 5813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:01.284528 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:01.284770 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:01.289680 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:01.289700 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:01.289733 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:01.290500 5813 server_base.cc:1034] running on GCE node
I20260928 17:52:01.290784 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:01.291096 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:01.292253 5813 hybrid_clock.cc:648] HybridClock initialized: now 1790617921292232 us; error 42 us; skew 500 ppm
I20260928 17:52:01.293507 5813 webserver.cc:460] Webserver started at http://127.0.98.62:37337/ using document root <none> and password file <none>
I20260928 17:52:01.293767 5813 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:01.293838 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:01.293989 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:01.295032 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "330a12203d87d607f322b36d608c48f3"
server_key_iv: "6229d71aa27b8be260baefd3bf46ab54"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.295390 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "330a12203d87d607f322b36d608c48f3"
server_key_iv: "6229d71aa27b8be260baefd3bf46ab54"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.296723 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:01.297420 5827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:01.297578 5813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:01.297667 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9"
format_stamp: "Formatted at 2026-09-28 17:52:01 on dist-test-slave-zqlt"
server_key: "330a12203d87d607f322b36d608c48f3"
server_key_iv: "6229d71aa27b8be260baefd3bf46ab54"
server_key_version: "encryptionkey@0"
I20260928 17:52:01.297744 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:01.304036 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:01.304328 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:01.304471 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:01.308269 5813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40569
I20260928 17:52:01.308311 5879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40569 every 8 connection(s)
I20260928 17:52:01.308616 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:01.309449 5880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:01.311686 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9: Bootstrap starting.
I20260928 17:52:01.312350 5880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:01.312660 5880 log.cc:824] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:01.313337 5880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9: No bootstrap required, opened a new log
I20260928 17:52:01.313530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5813
I20260928 17:52:01.313644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:01.313819 392 external_mini_cluster.cc:1428] Setting key 1920380a17adfc2dd90899474aa662d9
I20260928 17:52:01.315657 5880 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260928 17:52:01.315841 5880 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:01.315881 5880 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5954134e0d419ab3ba2e44fd4f0dc9, State: Initialized, Role: FOLLOWER
I20260928 17:52:01.316006 5880 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260928 17:52:01.316113 5880 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:01.316145 5880 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:01.316198 5880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:01.316943 5880 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260928 17:52:01.317052 5880 leader_election.cc:302] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5954134e0d419ab3ba2e44fd4f0dc9; no voters:
I20260928 17:52:01.317196 5880 leader_election.cc:288] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:01.317258 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:01.317401 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [term 1 LEADER]: Becoming Leader. State: Replica: ed5954134e0d419ab3ba2e44fd4f0dc9, State: Running, Role: LEADER
I20260928 17:52:01.317435 5880 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:01.317487 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } }
I20260928 17:52:01.317809 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } } }
I20260928 17:52:01.317912 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:01.318137 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5954134e0d419ab3ba2e44fd4f0dc9. Latest consensus state: current_term: 1 leader_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5954134e0d419ab3ba2e44fd4f0dc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40569 } } }
I20260928 17:52:01.318207 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:01.318320 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:01.318753 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:01.321135 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:01.314355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.0.98.62:40569)
0928 17:52:01.314574 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:01.314583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:01.314616 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:52:01.315610 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:01.315619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:01.315702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:01.315859 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:01.316062 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.317824 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.318992 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.319562 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.319655 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.320472 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:01.320475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:01.320475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:01.320644 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:01.320675 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:01.320686 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:01.320739 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:01.320836 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:52:01.320921 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260928 17:52:01.322044 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:52:01.326942 5892 catalog_manager.cc:1347] Generated new cluster ID: 810c95c1611d4074b25ad2f31c321050
I20260928 17:52:01.327003 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:01.336386 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:01.336906 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:01.349013 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5954134e0d419ab3ba2e44fd4f0dc9: Generated new TSK 0
I20260928 17:52:01.349218 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:52:01.414038 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:01.355496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.98.62:40569)
0928 17:52:01.355558 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:01.355560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:01.355669 (+ 109us) server_negotiation.cc:407] Connection header received
0928 17:52:01.355747 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:01.355748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:01.355760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:01.355846 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 17:52:01.355990 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.356565 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.357376 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.357587 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.357659 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.357862 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 17:52:01.413757 (+ 55895us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 17:52:01.413947 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42818: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 17:52:01.414119 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42818: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 17:52:01.414129 5909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:01.355435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40569 (local address 127.0.0.1:42818)
0928 17:52:01.355599 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:01.355609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:01.355685 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:01.355845 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:01.355846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:01.355848 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 17:52:01.355906 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:01.355909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.356647 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.356649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:01.357276 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:01.357277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:01.357677 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:01.357684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:01.357745 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:01.357763 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 17:52:01.413900 (+ 56137us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 17:52:01.414023 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260928 17:52:01.414438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5813
2026-09-28T17:52:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (241 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:01 dist-test-slave-zqlt krb5kdc[5921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:01 dist-test-slave-zqlt krb5kdc[5921](info): set up 2 sockets
Sep 28 17:52:01 dist-test-slave-zqlt krb5kdc[5921](info): commencing operation
krb5kdc: starting...
W20260928 17:52:02.616343 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:02 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:52:02.730568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:02.810271 5934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:02.810525 5934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:02.810562 5934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:02.811720 5934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:02.811800 5934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:02.811822 5934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:02.811843 5934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:02.811872 5934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:02.813468 5934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:02.813772 5934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:02.814016 5934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:02.819072 5939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:02.819100 5940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:02.819196 5934 server_base.cc:1034] running on GCE node
W20260928 17:52:02.819078 5942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:02.819689 5934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:02.820065 5934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:02.821271 5934 hybrid_clock.cc:648] HybridClock initialized: now 1790617922821259 us; error 33 us; skew 500 ppm
Sep 28 17:52:02 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:02.824013 5934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:52:02.824514 5934 webserver.cc:460] Webserver started at http://127.0.98.62:39569/ using document root <none> and password file <none>
I20260928 17:52:02.824740 5934 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:02.824784 5934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:02.824940 5934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:02.826088 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "0946e7a6be37468daea1e46588c17a4f"
format_stamp: "Formatted at 2026-09-28 17:52:02 on dist-test-slave-zqlt"
server_key: "93d18267b0c0d1404c729fa5fa66c113"
server_key_iv: "f8690cd24d8e064f4abda2ea11ce8867"
server_key_version: "encryptionkey@0"
I20260928 17:52:02.826423 5934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "0946e7a6be37468daea1e46588c17a4f"
format_stamp: "Formatted at 2026-09-28 17:52:02 on dist-test-slave-zqlt"
server_key: "93d18267b0c0d1404c729fa5fa66c113"
server_key_iv: "f8690cd24d8e064f4abda2ea11ce8867"
server_key_version: "encryptionkey@0"
I20260928 17:52:02.827785 5934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:02.828572 5949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:02.828732 5934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:02.828832 5934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "0946e7a6be37468daea1e46588c17a4f"
format_stamp: "Formatted at 2026-09-28 17:52:02 on dist-test-slave-zqlt"
server_key: "93d18267b0c0d1404c729fa5fa66c113"
server_key_iv: "f8690cd24d8e064f4abda2ea11ce8867"
server_key_version: "encryptionkey@0"
I20260928 17:52:02.828892 5934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:02.841790 5934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:02.843915 5934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:02.844034 5934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:02.847879 5934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34535
I20260928 17:52:02.847951 6001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34535 every 8 connection(s)
I20260928 17:52:02.848483 5934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:02.849190 6002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:02.851498 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f: Bootstrap starting.
I20260928 17:52:02.852133 6002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:02.852450 6002 log.cc:824] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:02.853133 6002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f: No bootstrap required, opened a new log
I20260928 17:52:02.854697 6002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260928 17:52:02.854844 6002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:02.854894 6002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0946e7a6be37468daea1e46588c17a4f, State: Initialized, Role: FOLLOWER
I20260928 17:52:02.855089 6002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260928 17:52:02.855186 6002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:02.855247 6002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:02.855312 6002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:02.855504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 5934
I20260928 17:52:02.855609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:02.855799 392 external_mini_cluster.cc:1428] Setting key b9fba84d9aeafb6a6658b58fd04ceb39
I20260928 17:52:02.856665 6002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260928 17:52:02.856777 6002 leader_election.cc:302] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0946e7a6be37468daea1e46588c17a4f; no voters:
I20260928 17:52:02.856940 6002 leader_election.cc:288] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:02.857028 6007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:02.857188 6007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [term 1 LEADER]: Becoming Leader. State: Replica: 0946e7a6be37468daea1e46588c17a4f, State: Running, Role: LEADER
I20260928 17:52:02.857230 6002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:02.857309 6007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260928 17:52:02.857674 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0946e7a6be37468daea1e46588c17a4f. Latest consensus state: current_term: 1 leader_uuid: "0946e7a6be37468daea1e46588c17a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } } }
I20260928 17:52:02.857764 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:02.857928 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0946e7a6be37468daea1e46588c17a4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0946e7a6be37468daea1e46588c17a4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } } }
I20260928 17:52:02.857997 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:02.858111 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:02.858598 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:02.860260 6015 catalog_manager.cc:1347] Generated new cluster ID: 6c597a212ddb4b1ebf08d61cc985fb70
I20260928 17:52:02.860312 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:52:02 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:02.867359 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:02.856387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.98.62:34535)
0928 17:52:02.856605 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:02.856616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:02.856658 (+ 42us) server_negotiation.cc:407] Connection header received
0928 17:52:02.856794 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:02.856803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:02.856864 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:02.857024 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:02.858121 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:02.859514 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:02.860636 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:02.861066 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:02.861199 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:02.863529 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:02.863533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:02.863533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:02.863546 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:02.865787 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:02.866191 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:02.866194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:02.866194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:02.866297 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:02.866505 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:02.866506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:02.866506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:02.866753 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:02.866946 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:02.867013 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 17:52:02.867163 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260928 17:52:02.882418 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:02.883050 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:02.890183 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0946e7a6be37468daea1e46588c17a4f: Generated new TSK 0
I20260928 17:52:02.890362 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:52:02.916843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:02.998824 6028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:02.999022 6028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:02.999044 6028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:03.000176 6028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:03.000214 6028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:03.000239 6028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:03.001732 6028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:03.001909 6028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:03.002068 6028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:03.006989 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.006996 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.006997 6036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:03.007295 6028 server_base.cc:1034] running on GCE node
I20260928 17:52:03.007483 6028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:03.007690 6028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:03.008827 6028 hybrid_clock.cc:648] HybridClock initialized: now 1790617923008817 us; error 32 us; skew 500 ppm
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.011416 6028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:52:03.011889 6028 webserver.cc:460] Webserver started at http://127.0.98.1:39227/ using document root <none> and password file <none>
I20260928 17:52:03.012074 6028 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:03.012182 6028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:03.012285 6028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:03.013298 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "17b073c1e6784731b88f4617a32daa98"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "2bbc6d6f3d6862a5908d647dc90447ad"
server_key_iv: "4a72604d207a212d34abb495324c8163"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.013600 6028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17b073c1e6784731b88f4617a32daa98"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "2bbc6d6f3d6862a5908d647dc90447ad"
server_key_iv: "4a72604d207a212d34abb495324c8163"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.014856 6028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:52:03.015558 6043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.015726 6028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:03.015836 6028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "17b073c1e6784731b88f4617a32daa98"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "2bbc6d6f3d6862a5908d647dc90447ad"
server_key_iv: "4a72604d207a212d34abb495324c8163"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.015913 6028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:03.030553 6028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:03.033138 6028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:03.033321 6028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:03.033576 6028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:03.033898 6028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:03.033952 6028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.033988 6028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:03.034015 6028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.039994 6028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33725
I20260928 17:52:03.040040 6156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33725 every 8 connection(s)
I20260928 17:52:03.040382 6028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:03.041585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6028
I20260928 17:52:03.041669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:03.041896 392 external_mini_cluster.cc:1428] Setting key 019647451742488fbaa74e57e32e6d87
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:52:03.051055 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.041085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47419 (local address 127.0.98.62:34535)
0928 17:52:03.041134 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.041137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.041324 (+ 187us) server_negotiation.cc:407] Connection header received
0928 17:52:03.042426 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.042427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.042440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.042494 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.043143 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.043759 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.044839 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.045191 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.045269 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.047972 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.047974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.047974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.047984 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.049679 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.050100 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.050102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.050102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.050163 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.050429 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.050430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.050431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.050583 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.050660 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.050915 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.050987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:52:03.051225 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.041052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.1:47419)
0928 17:52:03.041208 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:03.041250 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:03.042314 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:03.042583 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:03.042587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:03.042851 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:03.043044 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.043048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.043861 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.043863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.044755 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.044758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.045285 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.045286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.045387 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.045975 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:03.045983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:03.047869 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:03.049786 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.049793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.049795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.050021 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.050240 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.050240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.050241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.050335 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.050654 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:03.050659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:03.050791 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 17:52:03.050879 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:52:03.050999 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 17:52:03.051839 6157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260928 17:52:03.051946 6157 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:03.052191 6157 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 17:52:03.052747 5966 ts_manager.cc:194] Registered new tserver with Master: 17b073c1e6784731b88f4617a32daa98 (127.0.98.1:33725)
I20260928 17:52:03.053390 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:52:03.066774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:03.147307 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:03.147522 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:03.147544 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:03.148772 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:03.148818 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:03.148844 6162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:03.150300 6162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:03.150492 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:03.150678 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:03.155613 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.155618 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.155712 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:03.155997 6162 server_base.cc:1034] running on GCE node
I20260928 17:52:03.156257 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:03.156548 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:03.157687 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790617923157676 us; error 46 us; skew 500 ppm
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.160408 6162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:52:03.160861 6162 webserver.cc:460] Webserver started at http://127.0.98.2:37643/ using document root <none> and password file <none>
I20260928 17:52:03.161091 6162 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:03.161149 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:03.161253 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:03.162288 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "b5990875c845435da21656ef330af0be"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "67f9f9932bd87609c2d9b06d6b7f232b"
server_key_iv: "d26f44e66d8765cad0e1bada5d83b3f7"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.162606 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b5990875c845435da21656ef330af0be"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "67f9f9932bd87609c2d9b06d6b7f232b"
server_key_iv: "d26f44e66d8765cad0e1bada5d83b3f7"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.163867 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:03.164672 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.164867 6162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:52:03.164973 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "b5990875c845435da21656ef330af0be"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "67f9f9932bd87609c2d9b06d6b7f232b"
server_key_iv: "d26f44e66d8765cad0e1bada5d83b3f7"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.165050 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:03.176757 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:03.178771 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:03.178939 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:03.179175 6162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:03.179486 6162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:03.179539 6162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.179579 6162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:03.179607 6162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.185467 6162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33461
I20260928 17:52:03.185509 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33461 every 8 connection(s)
I20260928 17:52:03.185818 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:03.191609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6162
I20260928 17:52:03.191717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:03.191910 392 external_mini_cluster.cc:1428] Setting key 4dd3d3b901f25c23e8f39a4741550901
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:03.195580 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.186552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57955 (local address 127.0.98.62:34535)
0928 17:52:03.186613 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.186615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.186831 (+ 216us) server_negotiation.cc:407] Connection header received
0928 17:52:03.187833 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.187833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.187843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.187902 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.188645 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.189135 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.190204 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.190392 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.190443 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.192714 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.192716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.192716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.192724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.194160 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.194603 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.194604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.194604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.194673 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.194926 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.194927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.194927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.195040 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.195118 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.195399 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.195489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:52:03.195787 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.186509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.2:57955)
0928 17:52:03.186680 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:03.186725 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:03.187729 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:03.187982 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:03.187985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:03.188349 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:03.188567 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.188572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.189224 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.189225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.190127 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.190129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.190488 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.190489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.190602 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.191063 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:03.191071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:03.192632 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:03.194292 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.194298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.194301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.194523 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.194747 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.194748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.194748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.194858 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.195109 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:03.195116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:03.195267 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 17:52:03.195378 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 17:52:03.195532 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260928 17:52:03.196346 6291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260928 17:52:03.196468 6291 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:03.196669 6291 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
I20260928 17:52:03.197121 5966 ts_manager.cc:194] Registered new tserver with Master: b5990875c845435da21656ef330af0be (127.0.98.2:33461)
I20260928 17:52:03.197496 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57955
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:52:03.214764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:03.298453 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:03.298774 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:03.298805 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:03.299984 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:03.300026 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:03.300053 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:03.301594 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:03.301798 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:03.301977 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:03.306988 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.306977 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:03.307016 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:03.307045 6296 server_base.cc:1034] running on GCE node
I20260928 17:52:03.307410 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:03.307654 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:03.308801 6296 hybrid_clock.cc:648] HybridClock initialized: now 1790617923308794 us; error 37 us; skew 500 ppm
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.311357 6296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:52:03.311852 6296 webserver.cc:460] Webserver started at http://127.0.98.3:34203/ using document root <none> and password file <none>
I20260928 17:52:03.312081 6296 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:03.312186 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:03.312364 6296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:03.313472 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ce93a8537944b37be1d08e465f97381"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "98611e5a9100196b668b576e4d25c201"
server_key_iv: "ee846f7516a67c181c075410756f15ca"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.313843 6296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ce93a8537944b37be1d08e465f97381"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "98611e5a9100196b668b576e4d25c201"
server_key_iv: "ee846f7516a67c181c075410756f15ca"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.315184 6296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:03.315928 6311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.316154 6296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:03.316277 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "5ce93a8537944b37be1d08e465f97381"
format_stamp: "Formatted at 2026-09-28 17:52:03 on dist-test-slave-zqlt"
server_key: "98611e5a9100196b668b576e4d25c201"
server_key_iv: "ee846f7516a67c181c075410756f15ca"
server_key_version: "encryptionkey@0"
I20260928 17:52:03.316365 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:03.331672 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:03.333796 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:03.333978 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:03.334240 6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:03.334569 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:03.334625 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.334666 6296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:03.334697 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:03.340662 6296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44345
I20260928 17:52:03.340710 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44345 every 8 connection(s)
I20260928 17:52:03.341084 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:03.349579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6296
I20260928 17:52:03.349725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:03.349934 392 external_mini_cluster.cc:1428] Setting key b24b3470bb2a33414ca17d44670fe82b
I20260928 17:52:03.351337 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.341909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34775 (local address 127.0.98.62:34535)
0928 17:52:03.341980 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.341981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.342161 (+ 180us) server_negotiation.cc:407] Connection header received
0928 17:52:03.343354 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.343356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.343369 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.343427 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.344224 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.344740 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.345878 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.346092 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.346181 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.348509 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.348511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.348511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.348520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.349647 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.350123 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.350125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.350125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.350192 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.350565 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.350566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.350566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.350736 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.350818 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.351151 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.351243 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:52:03.351557 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.341854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.3:34775)
0928 17:52:03.342016 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:03.342061 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:03.343215 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:03.343542 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:03.343546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:03.343822 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:03.344082 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.344136 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.344852 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.344853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.345779 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:03.345782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.346193 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.346193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:03.346339 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.346853 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:03.346861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:03.348409 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:03.349753 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.349761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.349763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.350056 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.350328 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:03.350329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:03.350329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:03.350493 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:03.350806 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:03.350813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:03.350993 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 17:52:03.351128 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 17:52:03.351307 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260928 17:52:03.352036 6425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260928 17:52:03.352193 6425 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:03.352396 6425 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
I20260928 17:52:03.352784 5966 ts_manager.cc:194] Registered new tserver with Master: 5ce93a8537944b37be1d08e465f97381 (127.0.98.3:44345)
I20260928 17:52:03.353121 5966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34775
I20260928 17:52:03.361994 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/test-admin.keytab.
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.377775 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 17:52:03.379528 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:03.378715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.0.98.1:33725)
0928 17:52:03.378903 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.378908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.378919 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:03.378945 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.378947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.378961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.379036 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.379436 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53090: BlockingRecv error: recv got EOF from 127.0.0.1:53090 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
W20260928 17:52:03.379627 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.380859 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.381736 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:52:03.389611 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.382293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.0.98.1:33725)
0928 17:52:03.382373 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.382376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.382434 (+ 58us) server_negotiation.cc:407] Connection header received
0928 17:52:03.382475 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.382476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.382487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.382535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.382911 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.383629 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.384377 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.384691 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.384759 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.386917 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.386925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.386925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.386932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.388573 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.388930 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.388933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.388934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.388989 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.389159 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.389159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.389160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.389289 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.389394 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.389458 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.389520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 17:52:03.392004 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:03.391333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.0.98.1:33725)
0928 17:52:03.391420 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.391422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.391452 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:52:03.391505 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.391506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.391516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.391588 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.391948 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53102: BlockingRecv error: recv got EOF from 127.0.0.1:53102 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 17:52:03.392026 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.393043 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.393919 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:52:03.401003 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.394444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.0.98.1:33725)
0928 17:52:03.394499 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.394501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.394604 (+ 103us) server_negotiation.cc:407] Connection header received
0928 17:52:03.394625 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.394626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.394635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.394683 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.395033 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.395527 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.396311 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.396507 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.396568 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.398533 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.398534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.398535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.398541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.399944 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.400324 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.400325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.400326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.400380 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.400532 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.400532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.400533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.400616 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.400683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.400776 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.400837 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 17:52:03.402678 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:03.402098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.0.98.1:33725)
0928 17:52:03.402164 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.402166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.402230 (+ 64us) server_negotiation.cc:407] Connection header received
0928 17:52:03.402314 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.402315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.402325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.402382 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.402628 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53120: BlockingRecv error: recv got EOF from 127.0.0.1:53120 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 17:52:03.402855 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790617898245922-392-0/krb5kdc/krb5cc)
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.403883 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:03.404846 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[5921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 17:52:03.411340 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:03.405327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.0.98.1:33725)
0928 17:52:03.405368 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:03.405369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:03.405380 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:03.405450 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:03.405451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:03.405460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:03.405527 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:03.405884 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.406334 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.407157 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:03.407367 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:03.407444 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:03.409386 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:03.409387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:03.409388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:03.409393 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:03.410488 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.410785 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.410786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.410786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.410831 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:03.410996 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:03.410997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:03.410998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:03.411078 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:03.411157 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:03.411232 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:52:03.411284 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:52:03.411970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6028
I20260928 17:52:03.416795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6162
I20260928 17:52:03.421356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6296
I20260928 17:52:03.426276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 5934
2026-09-28T17:52:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1838 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[6445](info): set up 2 sockets
Sep 28 17:52:03 dist-test-slave-zqlt krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260928 17:52:04.476306 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:04 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 17:52:04.594745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:04.678151 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:04.678399 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:04.678422 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:04.679518 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:04.679553 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:04.679567 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:04.679579 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:04.679591 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:04.681190 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:04.681401 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:04.681581 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:04.686952 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:04.686939 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:04.686926 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:04.686971 6458 server_base.cc:1034] running on GCE node
I20260928 17:52:04.687343 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:04.687619 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:04.688818 6458 hybrid_clock.cc:648] HybridClock initialized: now 1790617924688799 us; error 45 us; skew 500 ppm
Sep 28 17:52:04 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:04.695384 6458 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 17:52:04.695916 6458 webserver.cc:460] Webserver started at http://127.0.98.62:34161/ using document root <none> and password file <none>
I20260928 17:52:04.696211 6458 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:04.696281 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:04.696435 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:04.697526 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "32b99c0e0ca04cf588be01236a671848"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "494f3a6345d162c60b3e05c4fa28a1d1"
server_key_iv: "1b19e5048aeba589bead23ebe74f0e1b"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.697906 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "32b99c0e0ca04cf588be01236a671848"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "494f3a6345d162c60b3e05c4fa28a1d1"
server_key_iv: "1b19e5048aeba589bead23ebe74f0e1b"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.699247 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:04.700008 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:04.700217 6458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:04.700320 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "32b99c0e0ca04cf588be01236a671848"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "494f3a6345d162c60b3e05c4fa28a1d1"
server_key_iv: "1b19e5048aeba589bead23ebe74f0e1b"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.700402 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:04.722252 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:04.724206 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:04.724365 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:04.728442 6458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34305
I20260928 17:52:04.728482 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34305 every 8 connection(s)
I20260928 17:52:04.728875 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:04.729837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6458
I20260928 17:52:04.729875 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:04.729933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:04.730144 392 external_mini_cluster.cc:1428] Setting key 636510496ffb48ec21142feed0028bfb
I20260928 17:52:04.733074 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Bootstrap starting.
I20260928 17:52:04.733812 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:04.734103 6526 log.cc:824] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:04.734813 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: No bootstrap required, opened a new log
Sep 28 17:52:04 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:04.737370 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } }
I20260928 17:52:04.737527 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:04.737577 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b99c0e0ca04cf588be01236a671848, State: Initialized, Role: FOLLOWER
I20260928 17:52:04.737726 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } }
I20260928 17:52:04.737790 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:04.737828 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:04.737874 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:04.738431 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } }
I20260928 17:52:04.738548 6526 leader_election.cc:302] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32b99c0e0ca04cf588be01236a671848; no voters:
I20260928 17:52:04.738749 6526 leader_election.cc:288] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:04.738780 6531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:04.738883 6531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 LEADER]: Becoming Leader. State: Replica: 32b99c0e0ca04cf588be01236a671848, State: Running, Role: LEADER
I20260928 17:52:04.739017 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:04.738972 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } }
I20260928 17:52:04.739323 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } }
I20260928 17:52:04.739396 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:04.739521 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b99c0e0ca04cf588be01236a671848. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } }
I20260928 17:52:04.739591 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
W20260928 17:52:04.740844 6547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:52:04.740901 6547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:52:04.741170 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:04.741339 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:04.741376 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:04.730675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.98.62:34305)
0928 17:52:04.730836 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:04.730844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:04.730869 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:52:04.731342 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:04.731349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:04.731396 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:04.731514 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:04.732346 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.733286 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.734365 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.734866 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.734962 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:04.737573 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:04.737577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:04.737577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:04.737590 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:04.739627 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:04.740060 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:04.740063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:04.740063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:04.740200 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:04.740398 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:04.740400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:04.740400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:04.740677 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:04.740838 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:04.741020 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 17:52:04.741169 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 17:52:04.742748 6536 catalog_manager.cc:1347] Generated new cluster ID: a0ca96e2187a4515b09ef65b3ac703c5
I20260928 17:52:04.742805 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:04.747272 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:04.747925 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:04.750855 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Generated new TSK 0
I20260928 17:52:04.751034 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 17:52:04.779280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:04.864136 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:04.864398 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:04.864456 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:04.865725 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:04.865803 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:04.865852 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:04.867437 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:04.867730 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:04.868471 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:04.873870 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:04.873870 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:04.873916 6552 server_base.cc:1034] running on GCE node
W20260928 17:52:04.873878 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:04.874348 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:04.874621 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:04.875789 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790617924875789 us; error 41 us; skew 500 ppm
Sep 28 17:52:04 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:04.878361 6552 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 17:52:04.878769 6552 webserver.cc:460] Webserver started at http://127.0.98.1:37919/ using document root <none> and password file <none>
I20260928 17:52:04.878957 6552 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:04.878998 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:04.879096 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:04.880200 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d175a3b1507455ebd5814f107b8ceeb"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "f106fbc464adc3b9c5bc9500cffa23aa"
server_key_iv: "672e9ee092116f90149f27b8ca0cfd46"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.880517 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d175a3b1507455ebd5814f107b8ceeb"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "f106fbc464adc3b9c5bc9500cffa23aa"
server_key_iv: "672e9ee092116f90149f27b8ca0cfd46"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.881783 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:04.882488 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:04.882725 6552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:04.882810 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "8d175a3b1507455ebd5814f107b8ceeb"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "f106fbc464adc3b9c5bc9500cffa23aa"
server_key_iv: "672e9ee092116f90149f27b8ca0cfd46"
server_key_version: "encryptionkey@0"
I20260928 17:52:04.882898 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:04.901787 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:04.903985 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:04.904170 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:04.904376 6552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:04.904682 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:04.904743 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:04.904788 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:04.904821 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:04.910496 6552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37301
I20260928 17:52:04.910573 6680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37301 every 8 connection(s)
I20260928 17:52:04.910905 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:04.914670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6552
I20260928 17:52:04.914754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:04.914961 392 external_mini_cluster.cc:1428] Setting key db2cd1ee4e87e993ef96bf2ae5d00980
Sep 28 17:52:04 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:04.921080 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:04.911718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40729 (local address 127.0.98.62:34305)
0928 17:52:04.911801 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:04.911804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:04.912002 (+ 198us) server_negotiation.cc:407] Connection header received
0928 17:52:04.912562 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:04.912563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:04.912573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:04.912628 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:04.913571 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.914068 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.915274 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.915541 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.915609 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:04.917911 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:04.917913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:04.917913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:04.917921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:04.919519 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:04.919911 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:04.919913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:04.919913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:04.919981 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:04.920272 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:04.920273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:04.920273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:04.920461 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:04.920546 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:04.920838 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 17:52:04.920944 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 17:52:04.921088 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:04.911687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:40729)
0928 17:52:04.911870 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:04.911908 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:04.912470 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:04.912695 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:04.912699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:04.913291 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:04.913487 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:04.913492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.914144 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.914145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:04.915062 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:04.915065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:04.915640 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:04.915641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:04.915822 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:04.916435 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:04.916443 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:04.917815 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:04.919610 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:04.919617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:04.919618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:04.919833 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:04.920052 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:04.920053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:04.920053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:04.920201 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:04.920538 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:04.920543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:04.920670 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 17:52:04.920771 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 17:52:04.920890 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260928 17:52:04.921701 6681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:04.921818 6681 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:04.922016 6681 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:04.922554 6490 ts_manager.cc:194] Registered new tserver with Master: 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:04.923172 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40729
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 17:52:04.940523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:05.021575 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.021806 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.021832 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.022948 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.022986 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.023012 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:05.024425 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:05.024672 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.024890 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.029819 6692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.029836 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.029836 6694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.030146 6686 server_base.cc:1034] running on GCE node
I20260928 17:52:05.030355 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.030604 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.031769 6686 hybrid_clock.cc:648] HybridClock initialized: now 1790617925031752 us; error 29 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.034333 6686 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 17:52:05.034814 6686 webserver.cc:460] Webserver started at http://127.0.98.2:34997/ using document root <none> and password file <none>
I20260928 17:52:05.035048 6686 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.035116 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.035286 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:05.036466 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "a3ca743e6a9e45948769e293533d35e2"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "4bfb6047a33e222e377e1d64b93dd37e"
server_key_iv: "1fb680e3256967af1eb5a0a0b85ef1b8"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.036849 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a3ca743e6a9e45948769e293533d35e2"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "4bfb6047a33e222e377e1d64b93dd37e"
server_key_iv: "1fb680e3256967af1eb5a0a0b85ef1b8"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.038213 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:05.038972 6701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.039173 6686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.039284 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "a3ca743e6a9e45948769e293533d35e2"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "4bfb6047a33e222e377e1d64b93dd37e"
server_key_iv: "1fb680e3256967af1eb5a0a0b85ef1b8"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.039363 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.053730 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.055750 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.055930 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.056290 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:05.056604 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:05.056638 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.056663 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:05.056675 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.062426 6686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33601
I20260928 17:52:05.062507 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33601 every 8 connection(s)
I20260928 17:52:05.062871 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:05.065299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6686
I20260928 17:52:05.065382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:05.065577 392 external_mini_cluster.cc:1428] Setting key 61d14a6d891408041d54374e9317f954
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:05.073155 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.063742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59277 (local address 127.0.98.62:34305)
0928 17:52:05.063819 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.063822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.064058 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:52:05.064545 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.064546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.064555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.064608 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.065616 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.066359 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.067416 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.067767 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.067815 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.070280 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.070281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.070282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.070290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.071722 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.072155 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.072157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.072158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.072230 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.072486 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.072487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.072487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.072608 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.072685 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.072977 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.073065 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:52:05.073336 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.063747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:59277)
0928 17:52:05.063932 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.063969 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.064457 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.064663 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.064666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.065257 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.065517 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.065522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.066461 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.066462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.067281 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.067284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.067916 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.067917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.068072 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.068731 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.068740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.070178 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.071818 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.071825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.071827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.072048 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.072298 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.072299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.072300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.072413 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.072675 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.072688 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.072832 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.072947 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.073105 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:52:05.073818 6815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:05.073930 6815 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:05.074131 6815 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:05.074563 6490 ts_manager.cc:194] Registered new tserver with Master: a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601)
I20260928 17:52:05.074956 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 17:52:05.089004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:05.170014 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.170221 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.170244 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.171334 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.171368 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.171393 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:05.172822 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:05.173031 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.173245 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.178153 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.178228 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.178184 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.178349 6820 server_base.cc:1034] running on GCE node
I20260928 17:52:05.178557 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.178797 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.179955 6820 hybrid_clock.cc:648] HybridClock initialized: now 1790617925179941 us; error 26 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.182341 6820 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 17:52:05.182780 6820 webserver.cc:460] Webserver started at http://127.0.98.3:41763/ using document root <none> and password file <none>
I20260928 17:52:05.183004 6820 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.183070 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.183207 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:05.184343 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "e7d5ac0fbe78450c9742c5faddb9f213"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2283fa4734a0287afb2de989d66acdb"
server_key_iv: "5007f35e6ad8539cfcf41b99585b21cf"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.184708 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e7d5ac0fbe78450c9742c5faddb9f213"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2283fa4734a0287afb2de989d66acdb"
server_key_iv: "5007f35e6ad8539cfcf41b99585b21cf"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.186110 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.186827 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.186991 6820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:05.187095 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "e7d5ac0fbe78450c9742c5faddb9f213"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2283fa4734a0287afb2de989d66acdb"
server_key_iv: "5007f35e6ad8539cfcf41b99585b21cf"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.187170 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.195772 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.197651 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.197821 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.198050 6820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:05.198352 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:05.198405 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.198446 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:05.198473 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.204267 6820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34983
I20260928 17:52:05.204324 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34983 every 8 connection(s)
I20260928 17:52:05.204638 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:05.213387 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.205399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42347 (local address 127.0.98.62:34305)
0928 17:52:05.205480 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.205482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.205657 (+ 175us) server_negotiation.cc:407] Connection header received
0928 17:52:05.206105 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.206106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.206115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.206166 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.207041 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.207525 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.208644 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.208841 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.208893 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.210912 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.210913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.210913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.210919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.212294 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.212640 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.212641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.212641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.212691 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.212892 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.212893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.212893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.213002 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.213052 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.213269 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.213325 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 17:52:05.213539 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.205367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:42347)
0928 17:52:05.205534 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.205569 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.206025 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.206221 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.206224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.206784 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.206968 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.206973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.207599 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.207599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.208573 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.208576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.208909 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.208910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.209008 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.209531 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.209539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.210838 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.212364 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.212369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.212371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.212574 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.212747 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.212747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.212748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.212839 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.213053 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.213057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.213162 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.213243 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.213353 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 17:52:05.213727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6820
I20260928 17:52:05.213820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:05.214001 6949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:05.214032 392 external_mini_cluster.cc:1428] Setting key f802158e596028ad8598f4b2b74c86f1
I20260928 17:52:05.214113 6949 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:05.214323 6949 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:05.214727 6490 ts_manager.cc:194] Registered new tserver with Master: e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983)
I20260928 17:52:05.215195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:52:05.215365 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42347
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 17:52:05.239288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32983
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34305,127.0.98.61:32983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:05.323974 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.324226 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.324250 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.325356 6954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:05.325397 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.325413 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.325428 6954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:05.325441 6954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:05.326815 6954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34305,127.0.98.61:32983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
W20260928 17:52:05.326911 6954 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34305,127.0.98.61:32983'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 17:52:05.327018 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.327193 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.331846 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.331943 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.331846 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.332063 6954 server_base.cc:1034] running on GCE node
I20260928 17:52:05.332310 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.332608 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.333755 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790617925333741 us; error 29 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.336361 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 17:52:05.336820 6954 webserver.cc:460] Webserver started at http://127.0.98.61:45269/ using document root <none> and password file <none>
I20260928 17:52:05.337056 6954 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.337105 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.337268 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:05.338405 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/instance:
uuid: "ba4a662729254e25b0c2de598f47685f"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "7d677c940a2622eeb957dee7ce1c4d24"
server_key_iv: "8b1465b49c43cf15fcc6f8c4dbf7c9b3"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.338809 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal/instance:
uuid: "ba4a662729254e25b0c2de598f47685f"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "7d677c940a2622eeb957dee7ce1c4d24"
server_key_iv: "8b1465b49c43cf15fcc6f8c4dbf7c9b3"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.340438 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 17:52:05.341239 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.341416 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:05.341521 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "ba4a662729254e25b0c2de598f47685f"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "7d677c940a2622eeb957dee7ce1c4d24"
server_key_iv: "8b1465b49c43cf15fcc6f8c4dbf7c9b3"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.341606 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.368680 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.370649 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.370829 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.375460 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32983
I20260928 17:52:05.375510 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32983 every 8 connection(s)
I20260928 17:52:05.375906 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
I20260928 17:52:05.376941 7022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:05.378464 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:05.384791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 6954
I20260928 17:52:05.388129 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.379197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.0.98.62:34305)
0928 17:52:05.379298 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.379300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.379432 (+ 132us) server_negotiation.cc:407] Connection header received
0928 17:52:05.379920 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.379921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.379930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.379984 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.380719 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.381207 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.382297 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.382552 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.382609 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.384903 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.384905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.384905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.384915 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.386648 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.387072 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.387073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.387074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.387139 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.387433 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.387434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.387434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.387588 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.387641 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.387961 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.388023 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 17:52:05.388397 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.379132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39048)
0928 17:52:05.379297 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.379348 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.379831 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.380061 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.380064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.380402 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.380638 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.380643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.381289 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.381290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.382212 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.382215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.382632 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.382632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.382737 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.383340 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.383349 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.384790 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.386744 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.386751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.386754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.386982 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.387208 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.387209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.387209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.387361 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.387655 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.387660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.387827 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.387940 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.388118 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260928 17:52:05.389027 7022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 17:52:05.394193 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.385297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.0.98.61:32983)
0928 17:52:05.385476 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.385481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.385506 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:52:05.385528 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.385530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.385542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.385632 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.386477 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.387206 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.388221 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.388657 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.388719 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.391255 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.391263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.391263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.391270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.392930 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.393321 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.393324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.393324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.393385 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.393595 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.393597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.393597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.393753 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.393918 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.394000 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.394079 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 17:52:05.396483 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.389309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:52980)
0928 17:52:05.389500 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.389511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.389561 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.389693 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.389694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.389861 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.389897 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.389898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.390528 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.390529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.391086 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.391088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.391565 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.391566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.391632 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.391923 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.391927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.393162 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.395435 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.395436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.395436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.395591 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.395801 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.395802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.395802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.395854 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.396118 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.396119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.396157 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.396252 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.396349 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260928 17:52:05.396559 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.389364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.98.61:32983)
0928 17:52:05.389513 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.389515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.389547 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:52:05.389616 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.389616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.389626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.389691 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.389982 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.390451 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.391170 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.391499 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.391567 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.393259 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.393261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.393261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.393267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.395344 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.395677 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.395678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.395679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.395730 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.395925 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.395925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.395926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.396035 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.396126 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.396274 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.396400 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260928 17:52:05.397890 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Bootstrap starting.
I20260928 17:52:05.398466 7022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:05.398689 7022 log.cc:824] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:05.399327 7022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: No bootstrap required, opened a new log
I20260928 17:52:05.400759 7022 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } }
I20260928 17:52:05.400883 7022 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:05.400905 7022 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4a662729254e25b0c2de598f47685f, State: Initialized, Role: FOLLOWER
I20260928 17:52:05.401031 7022 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } }
I20260928 17:52:05.401330 7022 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:05.401299 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } } }
I20260928 17:52:05.401399 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:52:05.402912 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:52:05.402992 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:52:05.404578 6954 master_runner.cc:428] Detected that this master ba4a662729254e25b0c2de598f47685f is joining an existing cluster
I20260928 17:52:05.404635 6954 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:32983
I20260928 17:52:05.412492 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.405552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.0.98.62:34305)
0928 17:52:05.405594 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.405596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.405963 (+ 367us) server_negotiation.cc:407] Connection header received
0928 17:52:05.406022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.406023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.406032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.406101 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.406541 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.407049 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.407805 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.408010 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.408090 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.409149 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.409151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.409151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.409159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.410803 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.411224 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.411226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.411226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.411293 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.411549 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.411551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.411551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.411705 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.411753 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.412297 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.412387 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:52:05.413024 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.405528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39056)
0928 17:52:05.405891 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.405902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.405962 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.406120 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.406122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.406397 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.406457 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.406459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.407129 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.407130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.407693 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.407695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.408123 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.408125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.408192 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.408683 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.408688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.409025 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.410891 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.410892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.410892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.411113 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.411372 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.411373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.411373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.411441 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.411802 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.411803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.411858 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.412625 (+ 767us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.412740 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":73,"threads_started":1}
I20260928 17:52:05.413643 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.405790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.0.98.61:32983)
0928 17:52:05.405824 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.405825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.406100 (+ 275us) server_negotiation.cc:407] Connection header received
0928 17:52:05.406162 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.406163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.406173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.406242 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.406623 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.407364 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.408494 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.408827 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.408867 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.410132 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.410134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.410134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.410141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.411807 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.412199 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.412200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.412200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.412251 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.412498 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.412498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.412499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.412633 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.412678 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.413429 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.413548 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":17}
I20260928 17:52:05.413654 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.405768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:52996)
0928 17:52:05.406053 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.406061 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.406117 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.406246 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.406246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.406471 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.406536 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.406538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.407477 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.407478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.408404 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.408406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.408933 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.408934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.408989 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.409402 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.409406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.409791 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.411900 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.411900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.411901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.412122 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.412319 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.412319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.412320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.412424 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.413059 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.413060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.413307 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.413385 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.413487 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":65,"threads_started":1}
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 17:52:05.423561 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.416393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.0.98.61:32983)
0928 17:52:05.416428 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.416429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.416597 (+ 168us) server_negotiation.cc:407] Connection header received
0928 17:52:05.416690 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.416691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.416702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.416759 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.417189 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.417649 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.418454 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.418659 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.418748 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.420989 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.420991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.420991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.420997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.422316 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.422730 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.422731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.422732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.422787 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.422988 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.422989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.422990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.423101 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.423175 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.423378 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.423459 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:52:05.423578 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.416358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:53008)
0928 17:52:05.416492 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.416535 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.416623 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.416765 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.416769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.417058 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.417108 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.417110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.417734 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.417735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.418354 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.418356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.418750 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.418751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.418838 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.419313 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.419318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.420889 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.422401 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.422409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.422412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.422628 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.422854 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.422855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.422855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.422923 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.423179 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.423186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.423255 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.423340 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.423470 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 17:52:05.424470 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:32983: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ba4a662729254e25b0c2de598f47685f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 32983
}
attrs {
promote: true
}
}
dest_uuid: "32b99c0e0ca04cf588be01236a671848"
cas_config_opid_index: -1
I20260928 17:52:05.424815 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } }
I20260928 17:52:05.425347 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba4a662729254e25b0c2de598f47685f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.425643 6996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:05.426512 6996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 17:52:05.426834 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:05.427609 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:32983
I20260928 17:52:05.427706 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.427788 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:05.427856 6954 master.cc:537] Master@127.0.98.61:32983 shutting down...
I20260928 17:52:05.429453 6996 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba4a662729254e25b0c2de598f47685f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.432299 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.432536 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b99c0e0ca04cf588be01236a671848. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.432823 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.432837 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.433529 6954 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Raft consensus shutting down.
I20260928 17:52:05.433650 6954 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Raft consensus is shut down!
I20260928 17:52:05.439162 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.439314 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.439409 6954 tablet_replica.cc:330] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: stopping tablet replica
I20260928 17:52:05.453032 6954 master.cc:552] Master@127.0.98.61:32983 shutdown complete.
I20260928 17:52:05.453107 6954 master_runner.cc:305] Clearing existing system tablet
I20260928 17:52:05.453250 6954 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.453300 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.454061 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.454640 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.454818 6954 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.454883 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.454909 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "ba4a662729254e25b0c2de598f47685f"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "7d677c940a2622eeb957dee7ce1c4d24"
server_key_iv: "8b1465b49c43cf15fcc6f8c4dbf7c9b3"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.455694 6954 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:05.457270 6954 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 17:52:05.457391 6954 log.cc:1196] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 17:52:05.457649 6954 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Deleting consensus metadata
I20260928 17:52:05.457782 6954 master_runner.cc:315] Copying system tablet from 127.0.98.62:34305
I20260928 17:52:05.458402 6954 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34305
I20260928 17:52:05.465283 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.458719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39064)
0928 17:52:05.458860 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.458872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.458942 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.459092 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.459093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.459269 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.459324 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.459326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.460299 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.460300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.460847 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.460848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.461340 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.461341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.461401 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.461927 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.461932 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.462243 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.464208 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.464209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.464209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.464389 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.464618 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.464619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.464619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.464674 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.464966 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.464967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.465011 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.465070 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.465188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260928 17:52:05.465296 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.458794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.98.62:34305)
0928 17:52:05.458851 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.458854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.458930 (+ 76us) server_negotiation.cc:407] Connection header received
0928 17:52:05.459014 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.459015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.459025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.459080 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.459405 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.460217 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.460934 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.461244 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.461320 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.462336 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.462338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.462338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.462346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.463999 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.464470 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.464478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.464478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.464545 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.464755 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.464756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.464757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.464894 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.464962 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.465117 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.465200 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 17:52:05.465656 6510 tablet_copy_service.cc:140] P 32b99c0e0ca04cf588be01236a671848: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ba4a662729254e25b0c2de598f47685f ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39064)
I20260928 17:52:05.465766 6510 tablet_copy_service.cc:161] P 32b99c0e0ca04cf588be01236a671848: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ba4a662729254e25b0c2de598f47685f at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39064: session id = ba4a662729254e25b0c2de598f47685f-00000000000000000000000000000000
I20260928 17:52:05.466415 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:52:05.467105 6954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 17:52:05.468489 6954 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: tablet copy: Starting download of 0 data blocks...
I20260928 17:52:05.468653 6954 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: tablet copy: Starting download of 1 WAL segments...
I20260928 17:52:05.469614 6954 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:52:05.470553 6510 tablet_copy_service.cc:342] P 32b99c0e0ca04cf588be01236a671848: Request end of tablet copy session ba4a662729254e25b0c2de598f47685f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39064
I20260928 17:52:05.470630 6510 tablet_copy_service.cc:433] P 32b99c0e0ca04cf588be01236a671848: ending tablet copy session ba4a662729254e25b0c2de598f47685f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ba4a662729254e25b0c2de598f47685f
I20260928 17:52:05.473163 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.474467 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.474570 6954 server_base.cc:1034] running on GCE node
W20260928 17:52:05.474486 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.474467 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.474896 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.475039 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.476141 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790617925476136 us; error 25 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.477641 6954 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 17:52:05.477907 6954 webserver.cc:460] Webserver started at http://127.0.98.61:37131/ using document root <none> and password file <none>
I20260928 17:52:05.478041 6954 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.478094 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.478722 6954 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:05.479234 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.479357 6954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.479403 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "ba4a662729254e25b0c2de598f47685f"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "7d677c940a2622eeb957dee7ce1c4d24"
server_key_iv: "8b1465b49c43cf15fcc6f8c4dbf7c9b3"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.479460 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.486416 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.487964 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.491031 6954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32983
I20260928 17:52:05.491048 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32983 every 8 connection(s)
I20260928 17:52:05.491365 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
I20260928 17:52:05.492229 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:52:05.492745 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Bootstrap starting.
I20260928 17:52:05.494959 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:52:05.495177 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Bootstrap complete.
I20260928 17:52:05.495357 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } }
I20260928 17:52:05.495446 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba4a662729254e25b0c2de598f47685f, State: Initialized, Role: LEARNER
I20260928 17:52:05.495484 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } }
I20260928 17:52:05.496035 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:05.496021 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } }
I20260928 17:52:05.496275 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.497859 7145 catalog_manager.cc:1259] Loaded cluster ID: a0ca96e2187a4515b09ef65b3ac703c5
I20260928 17:52:05.497912 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: loading cluster ID for follower catalog manager: success
I20260928 17:52:05.499178 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: acquiring CA information for follower catalog manager: success
I20260928 17:52:05.499586 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:52:05.503473 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.495776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.98.61:32983)
0928 17:52:05.496046 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.496049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.496059 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:05.496082 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.496082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.496093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.496978 (+ 885us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.497017 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.498463 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.499169 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.499382 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.499458 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.500698 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.500700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.500700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.500707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.502239 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.502634 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.502635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.502636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.502692 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.502899 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.502900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.502901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.503021 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.503163 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.503247 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.503329 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":223,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 17:52:05.527604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38311
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:05.612684 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.612877 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.612898 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.613951 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:05.613986 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.613999 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.614012 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:05.614024 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:05.615392 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:05.615571 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.615741 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:52:05.620448 7148 server_base.cc:1034] running on GCE node
W20260928 17:52:05.620496 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.620608 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.620446 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.620910 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.621245 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.622401 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790617925622395 us; error 37 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.624785 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 17:52:05.625141 7148 webserver.cc:460] Webserver started at http://127.0.98.60:41497/ using document root <none> and password file <none>
I20260928 17:52:05.625321 7148 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.625361 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.625449 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:05.626420 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/instance:
uuid: "d302a92aad304957a107161e9b64ba98"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2fd503efb2ce3b0ff7e33a1125ebb04"
server_key_iv: "123a1a1e55ac354408a526701609c2df"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.626773 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal/instance:
uuid: "d302a92aad304957a107161e9b64ba98"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2fd503efb2ce3b0ff7e33a1125ebb04"
server_key_iv: "123a1a1e55ac354408a526701609c2df"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.628059 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:05.628829 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.628994 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:05.629076 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "d302a92aad304957a107161e9b64ba98"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2fd503efb2ce3b0ff7e33a1125ebb04"
server_key_iv: "123a1a1e55ac354408a526701609c2df"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.629127 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.635569 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.637279 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.637382 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.641614 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38311
I20260928 17:52:05.641660 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38311 every 8 connection(s)
I20260928 17:52:05.642004 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
I20260928 17:52:05.642402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7148
I20260928 17:52:05.643182 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:05.644913 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:05.652805 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.643025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.0.98.60:38311)
0928 17:52:05.643212 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.643219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.643241 (+ 22us) server_negotiation.cc:407] Connection header received
0928 17:52:05.643753 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.643760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.643806 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.643946 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.644614 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.645669 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.646400 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.646806 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.646921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.649524 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.649530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.649531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.649537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.651188 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.651561 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.651565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.651565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.651654 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.651870 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.651871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.651871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.652179 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.652337 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.652447 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.652572 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260928 17:52:05.653378 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.645703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.62:34305)
0928 17:52:05.645749 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.645751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.645781 (+ 30us) server_negotiation.cc:407] Connection header received
0928 17:52:05.645863 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.645865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.645874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.645956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.646441 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.646923 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.647897 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.648125 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.648189 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.650575 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.650577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.650577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.650584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.651928 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.652361 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.652362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.652363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.652427 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.652678 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.652680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.652680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.652812 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.652919 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.653204 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.653285 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:52:05.653414 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.645522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39076)
0928 17:52:05.645684 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.645724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.645800 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.645977 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.645980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.646293 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.646358 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.646360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.647022 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.647023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.647788 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.647790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.648206 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.648207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.648278 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.648736 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.648747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.650490 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.652031 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.652038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.652040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.652280 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.652494 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.652495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.652495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.652560 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.652924 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.652931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.653015 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.653170 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.653301 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 17:52:05.654726 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 17:52:05.661327 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.655067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.0.98.61:32983)
0928 17:52:05.655134 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.655137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.655146 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:05.655176 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.655177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.655187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.655241 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.655508 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.656006 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.656739 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.656964 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.657053 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.658901 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.658902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.658903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.658911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.660192 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.660523 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.660524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.660524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.660589 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.660788 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.660789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.660789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.660911 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.660987 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.661146 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.661245 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:52:05.661319 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.654983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:53032)
0928 17:52:05.655034 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.655045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.655104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.655246 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.655247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.655385 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.655432 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.655434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.656107 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.656108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.656628 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.656629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.657051 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.657052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.657121 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.657539 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.657543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.658798 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.660273 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.660274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.660274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.660449 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.660656 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.660657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.660657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.660725 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.660977 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.660978 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.661023 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.661108 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.661214 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:52:05.661892 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 17:52:05.667470 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.662152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39076)
0928 17:52:05.662202 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.662212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.662249 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.662366 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.662367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.662473 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.662505 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.662506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.663079 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.663080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.663539 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.663540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.663847 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.663847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.663882 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.664363 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.664367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.665549 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.666737 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.666738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.666738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.666888 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.667035 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.667035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.667035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.667074 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.667249 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.667249 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.667279 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.667325 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.667392 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:52:05.667558 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.662212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.60:38311)
0928 17:52:05.662272 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.662274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.662281 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:05.662298 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.662298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.662306 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.662362 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.662559 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.663018 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.663604 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.663788 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.663838 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.665622 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.665623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.665623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.665629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.666672 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.666950 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.666951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.666951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.666996 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.667106 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.667107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.667107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.667190 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.667242 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.667340 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.667432 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":43}
I20260928 17:52:05.668876 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Bootstrap starting.
I20260928 17:52:05.669703 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:05.669991 7216 log.cc:824] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:05.670661 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: No bootstrap required, opened a new log
I20260928 17:52:05.672019 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } }
I20260928 17:52:05.672219 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:05.672277 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d302a92aad304957a107161e9b64ba98, State: Initialized, Role: FOLLOWER
I20260928 17:52:05.672410 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } }
I20260928 17:52:05.672652 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } } }
I20260928 17:52:05.672724 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:05.672745 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:52:05.675153 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:52:05.675209 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:52:05.677408 7148 master_runner.cc:428] Detected that this master d302a92aad304957a107161e9b64ba98 is joining an existing cluster
I20260928 17:52:05.677443 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:38311
I20260928 17:52:05.685284 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.60:38311)
0928 17:52:05.678682 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.678684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.678691 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:05.678737 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.678737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.678747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.678776 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.679277 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680012 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681085 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681357 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681394 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.682498 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.682500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.682500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.682507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.684051 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.684476 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.684477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.684478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.684530 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.684745 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.684746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.684746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.684857 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.684972 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.685058 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.685149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":18}
I20260928 17:52:05.685314 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39088)
0928 17:52:05.678602 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.678612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.678667 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.678833 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.678834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.679085 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.679191 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.679194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680131 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.680131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.680994 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.680996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681469 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.681529 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.682047 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.682052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.682388 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.684170 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.684172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.684172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.684387 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.684610 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.684611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.684611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.684665 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.684923 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.684924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.684963 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.685037 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.685201 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":77,"thread_start_us":78,"threads_started":1}
I20260928 17:52:05.700502 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.62:34305)
0928 17:52:05.678619 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.678621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.678934 (+ 313us) server_negotiation.cc:407] Connection header received
0928 17:52:05.679028 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.679029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.679039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.679069 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.679512 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680336 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681447 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681741 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681783 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.697408 (+ 15625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.697410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.697410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.697419 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.699048 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699449 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.699450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.699451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.699512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699850 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.699851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.699851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.699994 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.700060 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.700291 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.700399 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 17:52:05.700567 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.0.98.61:32983)
0928 17:52:05.678583 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.678585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.678711 (+ 126us) server_negotiation.cc:407] Connection header received
0928 17:52:05.678805 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.678806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.678817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.678847 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.679304 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680017 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681087 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681394 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681431 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.697489 (+ 16058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.697491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.697491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.697500 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.699327 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699714 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.699715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.699715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.699770 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699947 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.699948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.699949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.700160 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.700246 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.700400 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.700475 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 17:52:05.700980 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:53040)
0928 17:52:05.678657 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.678666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.678734 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.678906 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.678907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.679090 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.679224 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.679225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680131 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.680131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.680994 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.680996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681498 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.681560 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.696920 (+ 15360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.696925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.697382 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.699434 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.699436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.699644 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.699834 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.699834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.699878 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.700244 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.700245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.700290 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.700364 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.700472 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":7,"thread_start_us":49,"threads_started":1}
I20260928 17:52:05.700980 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.678750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39090)
0928 17:52:05.678876 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.678885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.678958 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.679138 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.679139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.679362 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.679421 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.679423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.680451 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.680452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.681346 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.681348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.681866 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.681867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.681930 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.696755 (+ 14825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.696762 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.697278 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.699174 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.699176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.699345 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.699579 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.699580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.699580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.699640 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.700061 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.700061 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.700158 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.700650 (+ 492us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.700756 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260928 17:52:05.704200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6552
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617924, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 17:52:05.710155 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.703099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39090)
0928 17:52:05.703147 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.703157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.703217 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.703359 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.703360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.703541 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.703588 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.703590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.704448 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.704449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.705142 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.705144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.705551 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.705552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.705616 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.706030 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.706035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.707202 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.709119 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.709120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.709121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.709292 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.709503 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.709504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.709504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.709560 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.709810 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.709811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.709857 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.709945 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.710052 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:52:05.710500 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.703154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.60:38311)
0928 17:52:05.703209 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.703211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.703219 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:05.703290 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.703291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.703300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.703355 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.703670 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.704365 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.705279 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.705480 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.705554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.707288 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.707289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.707290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.707297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.709012 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.709378 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.709379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.709379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.709432 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.709631 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.709632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.709632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.709743 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.709818 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.710065 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.710133 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:52:05.710987 6490 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:38311: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d302a92aad304957a107161e9b64ba98"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 38311
}
attrs {
promote: true
}
}
dest_uuid: "32b99c0e0ca04cf588be01236a671848"
cas_config_opid_index: 5
I20260928 17:52:05.711259 6490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } }
I20260928 17:52:05.711604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37301
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34305
--webserver_port=37919
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:52:05.711853 7241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d302a92aad304957a107161e9b64ba98 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.713092 7241 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:38311
I20260928 17:52:05.713191 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.713279 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:05.713308 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:05.713567 7148 master.cc:537] Master@127.0.98.60:38311 shutting down...
I20260928 17:52:05.714350 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 17:52:05.714782 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 17:52:05.716579 6475 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 17:52:05.717012 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 -> Peer d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311): Couldn't send request to peer d302a92aad304957a107161e9b64ba98. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 17:52:05.719409 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.712305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:53048)
0928 17:52:05.712389 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.712399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.712438 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.715149 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.715150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.715153 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:05.715205 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.715207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.716827 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.716828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.718039 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.718042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.719123 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.719124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.719210 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.719236 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.719290 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.719334 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 17:52:05.719833 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.712514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.98.61:32983)
0928 17:52:05.714654 (+ 2140us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.714657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.714667 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:05.714694 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.714695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.714735 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.714798 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:05.715296 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.716048 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.718139 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.718765 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.718834 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.718914 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.719718 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.719767 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2111}
I20260928 17:52:05.720137 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 17:52:05.720392 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 17:52:05.720860 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d302a92aad304957a107161e9b64ba98 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.721011 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:05.721138 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:05.721200 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: stopping tablet replica
I20260928 17:52:05.721925 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b99c0e0ca04cf588be01236a671848. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.722002 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.722035 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: true } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.722074 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.723263 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: attempting to promote NON_VOTER ba4a662729254e25b0c2de598f47685f to VOTER
I20260928 17:52:05.723479 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } }
I20260928 17:52:05.723932 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 LEARNER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 17:52:05.723968 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.723842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39096)
0928 17:52:05.723890 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.723932 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:52:05.724309 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.724220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39108)
0928 17:52:05.724259 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.724281 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 17:52:05.724378 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 -> Peer d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311): Couldn't send request to peer d302a92aad304957a107161e9b64ba98. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 17:52:05.724437 7241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 17:52:05.725054 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ba4a662729254e25b0c2de598f47685f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.725224 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ba4a662729254e25b0c2de598f47685f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.726054 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.726126 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:05.726223 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.726265 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:05.726500 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.726615 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:52:05.734424 7148 master.cc:552] Master@127.0.98.60:38311 shutdown complete.
I20260928 17:52:05.734483 7148 master_runner.cc:305] Clearing existing system tablet
I20260928 17:52:05.734617 7148 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.734661 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.735399 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:05.735950 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.736064 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.736361 7148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:52:05.736390 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "d302a92aad304957a107161e9b64ba98"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2fd503efb2ce3b0ff7e33a1125ebb04"
server_key_iv: "123a1a1e55ac354408a526701609c2df"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.737162 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:05.738655 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 17:52:05.738727 7148 log.cc:1196] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 17:52:05.738932 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Deleting consensus metadata
I20260928 17:52:05.739054 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:34305
I20260928 17:52:05.739689 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34305
I20260928 17:52:05.747328 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.740005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39106)
0928 17:52:05.740144 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.740156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.740232 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.740408 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.740409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.740640 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.740707 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.740710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.741623 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.741624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.742824 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.742827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.743316 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.743317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.743382 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.743905 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.743910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.744398 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.746234 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.746235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.746236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.746454 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.746667 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.746668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.746668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.746739 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.747009 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.747010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.747056 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.747126 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.747230 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":246,"threads_started":1}
I20260928 17:52:05.747342 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.740160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.98.62:34305)
0928 17:52:05.740217 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.740220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.740231 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:05.740326 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.740327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.740337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.740394 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.740784 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.741519 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.742913 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.743230 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.743308 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.744489 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.744490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.744491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.744498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.746130 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.746529 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.746531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.746531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.746594 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.746806 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.746807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.746808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.746941 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.747010 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.747165 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.747249 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:05.748150 6510 tablet_copy_service.cc:140] P 32b99c0e0ca04cf588be01236a671848: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d302a92aad304957a107161e9b64ba98 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39106)
I20260928 17:52:05.748242 6510 tablet_copy_service.cc:161] P 32b99c0e0ca04cf588be01236a671848: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d302a92aad304957a107161e9b64ba98 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39106: session id = d302a92aad304957a107161e9b64ba98-00000000000000000000000000000000
I20260928 17:52:05.748759 6510 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 17:52:05.749213 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 17:52:05.750432 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: tablet copy: Starting download of 0 data blocks...
I20260928 17:52:05.750569 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: tablet copy: Starting download of 1 WAL segments...
I20260928 17:52:05.751379 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 17:52:05.752364 6510 tablet_copy_service.cc:342] P 32b99c0e0ca04cf588be01236a671848: Request end of tablet copy session d302a92aad304957a107161e9b64ba98-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39106
I20260928 17:52:05.752421 6510 tablet_copy_service.cc:433] P 32b99c0e0ca04cf588be01236a671848: ending tablet copy session d302a92aad304957a107161e9b64ba98-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d302a92aad304957a107161e9b64ba98
I20260928 17:52:05.754935 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.756011 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.761148 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.761336 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.761369 7148 server_base.cc:1034] running on GCE node
I20260928 17:52:05.761592 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.761737 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.762866 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790617925762844 us; error 40 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.764358 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 17:52:05.764674 7148 webserver.cc:460] Webserver started at http://127.0.98.60:35461/ using document root <none> and password file <none>
I20260928 17:52:05.764811 7148 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.764846 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.765616 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:52:05.766222 7266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.766335 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.766379 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "d302a92aad304957a107161e9b64ba98"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2fd503efb2ce3b0ff7e33a1125ebb04"
server_key_iv: "123a1a1e55ac354408a526701609c2df"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.766440 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.789655 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.791678 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.794729 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38311
I20260928 17:52:05.794981 7318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38311 every 8 connection(s)
I20260928 17:52:05.795095 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
I20260928 17:52:05.796034 7319 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:52:05.796738 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Bootstrap starting.
I20260928 17:52:05.799701 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:52:05.799908 7319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: Bootstrap complete.
I20260928 17:52:05.800119 7319 raft_consensus.cc:348] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } }
I20260928 17:52:05.800213 7319 raft_consensus.cc:729] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d302a92aad304957a107161e9b64ba98, State: Initialized, Role: LEARNER
I20260928 17:52:05.800257 7319 consensus_queue.cc:260] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } }
I20260928 17:52:05.800427 7319 sys_catalog.cc:564] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:05.800426 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:05.800637 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: LEARNER
I20260928 17:52:05.802117 7333 catalog_manager.cc:1259] Loaded cluster ID: a0ca96e2187a4515b09ef65b3ac703c5
I20260928 17:52:05.802223 7333 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: loading cluster ID for follower catalog manager: success
I20260928 17:52:05.803483 7333 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: acquiring CA information for follower catalog manager: success
I20260928 17:52:05.803840 7333 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 17:52:05.813515 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.813714 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.813738 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.814949 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.814996 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.815022 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:05.816583 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37919
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:05.816825 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.817073 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:05.822175 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.822280 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.822175 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.822494 7242 server_base.cc:1034] running on GCE node
I20260928 17:52:05.822695 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.822942 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.824111 7242 hybrid_clock.cc:648] HybridClock initialized: now 1790617925824063 us; error 36 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.826694 7242 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 17:52:05.827231 7242 webserver.cc:460] Webserver started at http://127.0.98.1:37919/ using document root <none> and password file <none>
I20260928 17:52:05.827488 7242 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.827545 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.829124 7242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.829824 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.830077 7242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.830184 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "8d175a3b1507455ebd5814f107b8ceeb"
format_stamp: "Formatted at 2026-09-28 17:52:04 on dist-test-slave-zqlt"
server_key: "f106fbc464adc3b9c5bc9500cffa23aa"
server_key_iv: "672e9ee092116f90149f27b8ca0cfd46"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.830577 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.841231 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.843703 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.843880 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.844209 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:05.844565 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:05.844630 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.844669 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:05.844696 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.850732 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37301
I20260928 17:52:05.850822 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37301 every 8 connection(s)
I20260928 17:52:05.851183 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:05.859385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7242
I20260928 17:52:05.859571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6686
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 17:52:05.863301 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.852055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41071 (local address 127.0.98.62:34305)
0928 17:52:05.852137 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.852139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.852387 (+ 248us) server_negotiation.cc:407] Connection header received
0928 17:52:05.852979 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.852981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.852994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.853028 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.854368 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.855074 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856508 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856818 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856862 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.859591 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.859616 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.859617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.859625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.861532 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862060 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.862062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.862062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.862124 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862418 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.862419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.862420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.862582 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.862641 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.863036 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.863135 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:52:05.863802 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.852204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42617 (local address 127.0.98.60:38311)
0928 17:52:05.852297 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.852299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.852451 (+ 152us) server_negotiation.cc:407] Connection header received
0928 17:52:05.852955 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.852956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.852969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.853036 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.853993 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.854906 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856499 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856818 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856884 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.860489 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.860491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.860491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.860499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.862312 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862734 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.862736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.862736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.862800 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.863004 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.863004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.863005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.863144 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.863206 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.863579 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.863688 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260928 17:52:05.864035 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.851937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.98.1:42617)
0928 17:52:05.852389 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.852400 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.852867 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.853411 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.853412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.853685 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.853896 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.853901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.855009 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.855009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.856399 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.856401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856892 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.856956 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.857694 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.857705 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.860388 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.862403 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.862405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.862633 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.862876 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.862877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.862939 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.863232 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.863233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.863297 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.863366 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.863793 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":239,"thread_start_us":78,"threads_started":1}
I20260928 17:52:05.864632 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38311
I20260928 17:52:05.864774 7462 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:05.864984 7462 heartbeater.cc:511] Master 127.0.98.60:38311 requested a full tablet report, sending...
I20260928 17:52:05.865119 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.851937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:41071)
0928 17:52:05.852301 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.852310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.852854 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.853140 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.853145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.853686 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.854283 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.854285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.855151 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.855152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.856399 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.856401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856945 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.857005 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.857752 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.857763 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.859444 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.861639 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.861645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.861648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.861972 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.862194 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.862195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.862195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.862341 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.862689 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.862695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.862881 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.863648 (+ 767us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.864507 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":211,"thread_start_us":86,"threads_started":1}
I20260928 17:52:05.865341 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.851906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34327 (local address 127.0.98.61:32983)
0928 17:52:05.851951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.851953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.852406 (+ 453us) server_negotiation.cc:407] Connection header received
0928 17:52:05.852979 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.852981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.852994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.853028 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.855164 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.855626 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856486 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856692 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856779 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.861652 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:05.861654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:05.861654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:05.861663 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:05.863379 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.864224 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.864226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.864226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.864290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:05.864521 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:05.864522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:05.864522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:05.864898 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:05.864957 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:05.865141 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 17:52:05.865243 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:05.865700 7283 ts_manager.cc:194] Registered new tserver with Master: 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:05.865710 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.851937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.98.1:34327)
0928 17:52:05.852247 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.852294 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.852852 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.853140 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.853145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.853686 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.854271 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.854274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.855720 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.855721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.856395 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.856397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.856788 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.856789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.856956 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.857694 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.857705 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:05.861275 (+ 3570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:05.863463 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.863464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.863464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.863672 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.864363 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:05.864363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:05.864364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:05.864425 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:05.864990 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:05.864991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:05.865032 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 17:52:05.865447 (+ 415us) client_negotiation.cc:240] Negotiation successful
0928 17:52:05.865598 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260928 17:52:05.866420 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.61:32983
I20260928 17:52:05.866475 7464 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:05.866544 7464 heartbeater.cc:511] Master 127.0.98.61:32983 requested a full tablet report, sending...
I20260928 17:52:05.866922 7094 ts_manager.cc:194] Registered new tserver with Master: 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:05.867758 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:05.867817 7463 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:05.867887 7463 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:05.867975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33601
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34305
--webserver_port=34997
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:52:05.868208 6490 ts_manager.cc:194] Re-registered known tserver with Master: 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:05.868724 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41071
W20260928 17:52:05.951498 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:05.951738 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:05.951766 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:05.953088 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:05.953142 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:05.953172 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:05.954726 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34997
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:05.954983 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:05.955255 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:52:05.960950 7470 server_base.cc:1034] running on GCE node
W20260928 17:52:05.960922 7476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.960906 7475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:05.960913 7478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:05.961346 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:05.961638 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:05.962802 7470 hybrid_clock.cc:648] HybridClock initialized: now 1790617925962797 us; error 38 us; skew 500 ppm
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:05.965453 7470 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 17:52:05.965962 7470 webserver.cc:460] Webserver started at http://127.0.98.2:34997/ using document root <none> and password file <none>
I20260928 17:52:05.966213 7470 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:05.966336 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:05.967757 7470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:05.968397 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.968568 7470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:05.968673 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "a3ca743e6a9e45948769e293533d35e2"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "4bfb6047a33e222e377e1d64b93dd37e"
server_key_iv: "1fb680e3256967af1eb5a0a0b85ef1b8"
server_key_version: "encryptionkey@0"
I20260928 17:52:05.969036 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:05.982386 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:05.984248 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:05.984400 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:05.984622 7470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:05.984918 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:05.984972 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.985013 7470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:05.985040 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:05.991067 7470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33601
I20260928 17:52:05.991110 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33601 every 8 connection(s)
I20260928 17:52:05.991439 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:05.992831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7470
I20260928 17:52:05.992992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6820
I20260928 17:52:05.998706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34983
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34305
--webserver_port=41763
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 17:52:05 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617925, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 17:52:06.004854 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43529 (local address 127.0.98.62:34305)
0928 17:52:05.992253 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.992255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.992510 (+ 255us) server_negotiation.cc:407] Connection header received
0928 17:52:05.993130 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.993131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.993144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.993173 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.994759 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.995541 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997164 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997458 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997499 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.000827 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.000828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.000829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.000837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.003350 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.003779 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.003781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.003781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.003837 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.004044 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.004045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.004045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.004210 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.004271 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.004668 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.004753 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:06.004854 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59757 (local address 127.0.98.60:38311)
0928 17:52:05.992435 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.992438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.992690 (+ 252us) server_negotiation.cc:407] Connection header received
0928 17:52:05.993920 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.993921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.993932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.993991 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.994760 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.995541 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997165 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997483 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997520 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.000961 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.000963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.000963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.000972 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.002862 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.003657 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.003659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.003659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.003728 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.004044 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.004045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.004045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.004199 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.004271 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.004668 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.004761 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:52:06.005421 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.98.2:59757)
0928 17:52:05.992606 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.992614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.993842 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.994036 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.994037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.994351 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.994647 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.994653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.995676 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.995676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.997077 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.997078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997592 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.997745 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.998363 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.998371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.000854 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.002963 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.002971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.002974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.003567 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.003831 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.003832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.003832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.003966 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.004308 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.004315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.004487 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.004948 (+ 461us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.005174 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":101,"thread_start_us":72,"threads_started":1}
I20260928 17:52:06.005421 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:43529)
0928 17:52:05.992380 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.992418 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.992986 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.993330 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.993333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.994351 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.994647 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.994653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.995676 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.995676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.997062 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.997064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997594 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.997745 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:05.998475 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:05.998479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.000658 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.003473 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.003474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.003475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.003694 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.003903 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.003903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.003904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.003966 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.004308 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.004315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.004487 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.004972 (+ 485us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.005230 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":191,"thread_start_us":83,"threads_started":1}
I20260928 17:52:06.005919 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:06.006044 7600 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.006223 7600 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:06.006578 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37557 (local address 127.0.98.61:32983)
0928 17:52:05.992217 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:05.992219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:05.992536 (+ 317us) server_negotiation.cc:407] Connection header received
0928 17:52:05.993620 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:05.993621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:05.993634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:05.993675 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:05.996926 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997737 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.999309 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.999623 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.999696 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.001938 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.001939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.001941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.001974 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.004322 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.004800 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.004801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.004802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.004861 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.005479 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.005481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.005481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.005624 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.005703 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.006414 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.006498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:06.006580 6490 ts_manager.cc:194] Re-registered known tserver with Master: a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601)
I20260928 17:52:06.006634 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:05.992133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.98.2:37557)
0928 17:52:05.992375 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:05.992417 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:05.992993 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:05.993743 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:05.993744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:05.996707 (+ 2963us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:05.996831 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.996833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.997860 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.997861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.999223 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:05.999225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:05.999720 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:05.999721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:05.999794 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.000295 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:06.000300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.001850 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.004464 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.004465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.004465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.004713 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.005328 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.005329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.005330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.005401 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.006235 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.006236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.006289 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.006365 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.006478 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260928 17:52:06.007078 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43529
I20260928 17:52:06.007484 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.61:32983
I20260928 17:52:06.007534 7601 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.007670 7601 heartbeater.cc:511] Master 127.0.98.61:32983 requested a full tablet report, sending...
I20260928 17:52:06.008030 7094 ts_manager.cc:194] Registered new tserver with Master: a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601)
I20260928 17:52:06.008153 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38311
I20260928 17:52:06.008210 7599 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.008273 7599 heartbeater.cc:511] Master 127.0.98.60:38311 requested a full tablet report, sending...
I20260928 17:52:06.008589 7283 ts_manager.cc:194] Registered new tserver with Master: a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601)
W20260928 17:52:06.087795 7606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:06.088019 7606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:06.088045 7606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:06.089293 7606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:06.089340 7606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:06.089375 7606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:06.090770 7606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41763
--tserver_master_addrs=127.0.98.62:34305,127.0.98.61:32983,127.0.98.60:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:06.090958 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:06.091153 7606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:06.096076 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:06.096073 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:06.096272 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:06.096414 7606 server_base.cc:1034] running on GCE node
I20260928 17:52:06.096624 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:06.096869 7606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:06.098027 7606 hybrid_clock.cc:648] HybridClock initialized: now 1790617926098012 us; error 32 us; skew 500 ppm
Sep 28 17:52:06 dist-test-slave-zqlt krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:06.100675 7606 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 17:52:06.101172 7606 webserver.cc:460] Webserver started at http://127.0.98.3:41763/ using document root <none> and password file <none>
I20260928 17:52:06.101352 7606 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:06.101402 7606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:06.102782 7606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:52:06.103289 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:06.103448 7606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:06.103561 7606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "e7d5ac0fbe78450c9742c5faddb9f213"
format_stamp: "Formatted at 2026-09-28 17:52:05 on dist-test-slave-zqlt"
server_key: "d2283fa4734a0287afb2de989d66acdb"
server_key_iv: "5007f35e6ad8539cfcf41b99585b21cf"
server_key_version: "encryptionkey@0"
I20260928 17:52:06.103945 7606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:06.117610 7606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:06.119576 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:06.119750 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:06.120020 7606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:06.120380 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:06.120422 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:06.120452 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:06.120476 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:06.126502 7606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34983
I20260928 17:52:06.126549 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34983 every 8 connection(s)
I20260928 17:52:06.126911 7606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:52:06 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 17:52:06 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 17:52:06 dist-test-slave-zqlt krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617926, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:06.136758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7606
I20260928 17:52:06.137785 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35017 (local address 127.0.98.61:32983)
0928 17:52:06.127971 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.127973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.128008 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:52:06.128709 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.128711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.128722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.128774 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.129597 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.130348 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.132357 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.132595 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.132645 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.134527 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.134529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.134529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.134537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.136139 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.136641 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.136642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.136643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.136717 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137016 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.137017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.137017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.137170 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.137218 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.137588 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.137662 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:06.138383 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.98.3:35017)
0928 17:52:06.127883 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.127928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.128510 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.129182 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.129183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.129365 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:06.129486 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.129492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.131553 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.132241 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.132243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.132663 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.132664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.132732 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.133102 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:06.133107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.134438 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.136232 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.136237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.136239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.136550 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.136796 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.136796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.136797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.136931 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.137272 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.137278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.137433 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.137928 (+ 495us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.138106 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
I20260928 17:52:06.138479 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53405 (local address 127.0.98.60:38311)
0928 17:52:06.127764 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.127766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.128068 (+ 302us) server_negotiation.cc:407] Connection header received
0928 17:52:06.128628 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.128629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.128642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.128718 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.129597 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.130349 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131541 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.131840 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131911 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.135579 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.135580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.135580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.135586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.137256 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137681 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.137682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.137682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.137741 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137951 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.137951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.137952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.138065 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.138103 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.138294 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.138381 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 17:52:06.138899 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36631 (local address 127.0.98.62:34305)
0928 17:52:06.127980 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.127982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.128108 (+ 126us) server_negotiation.cc:407] Connection header received
0928 17:52:06.128826 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.128827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.128836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.128926 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.129580 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.130083 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131541 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.131840 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131911 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.135904 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.135905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.135906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.135913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.137684 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.138057 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.138058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.138058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.138115 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.138310 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.138311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.138311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.138455 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.138501 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.138740 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.138812 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:52:06.138886 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.61:32983
I20260928 17:52:06.138989 7737 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.139199 7737 heartbeater.cc:511] Master 127.0.98.61:32983 requested a full tablet report, sending...
I20260928 17:52:06.138909 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:36631)
0928 17:52:06.128024 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.128034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.128524 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.128949 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.128950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.129277 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:06.129485 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.129490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.130195 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.130196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.131450 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.131452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.131927 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.132059 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.132735 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:06.132744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.135804 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.137763 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.137764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.137972 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.138187 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.138187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.138188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.138243 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.138561 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.138562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.138605 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.138705 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.138807 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":33,"thread_start_us":104,"threads_started":1}
I20260928 17:52:06.139691 7094 ts_manager.cc:194] Registered new tserver with Master: e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983)
I20260928 17:52:06.139840 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34305
I20260928 17:52:06.139918 7736 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.139999 7736 heartbeater.cc:511] Master 127.0.98.62:34305 requested a full tablet report, sending...
I20260928 17:52:06.140066 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.127657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.98.3:53405)
0928 17:52:06.128012 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.128020 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.128509 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.128816 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.128820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.129290 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:06.129489 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.129492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.130423 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.130423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.131449 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.131452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.131985 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.131986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.132060 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.132821 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:06.132826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:06.135495 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:06.137355 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.137357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.137590 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.137815 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:06.137816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:06.137816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:06.137878 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:06.138152 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:06.138153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:06.138184 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.138269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.138366 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":41,"thread_start_us":108,"threads_started":1}
I20260928 17:52:06.140679 6490 ts_manager.cc:194] Re-registered known tserver with Master: e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983)
I20260928 17:52:06.140950 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.60:38311
I20260928 17:52:06.141007 7735 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:06.141079 7735 heartbeater.cc:511] Master 127.0.98.60:38311 requested a full tablet report, sending...
I20260928 17:52:06.141196 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36631
I20260928 17:52:06.141402 7283 ts_manager.cc:194] Registered new tserver with Master: e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983)
I20260928 17:52:06.147888 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.139185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.98.60:38311)
0928 17:52:06.139224 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.139226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.140757 (+ 1531us) server_negotiation.cc:407] Connection header received
0928 17:52:06.140838 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.140839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.140849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.140911 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.141937 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.142412 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.143142 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.143356 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.143443 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.145116 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.145117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.145118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.145125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.146742 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.147107 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.147108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.147108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.147171 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.147352 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.147353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.147353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.147465 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.147577 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.147680 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.147770 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:52:06.149260 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.139470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.0.98.61:32983)
0928 17:52:06.139532 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.139534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.140654 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 17:52:06.140916 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.140917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.140925 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.140953 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.142539 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.143253 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.144574 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.144844 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.144918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.145935 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.145937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.145937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.145945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.147785 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.148454 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.148455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.148455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.148505 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.148710 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.148711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.148711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.148842 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.148937 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.149055 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.149136 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:52:06.149798 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.139486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.0.98.62:34305)
0928 17:52:06.139562 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.139564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.140538 (+ 974us) server_negotiation.cc:407] Connection header received
0928 17:52:06.141923 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.141924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.141935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.141966 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.142499 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.143238 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.144348 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.144679 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.144762 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.146713 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.146715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.146715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.146722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.148539 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.148917 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.148917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.148918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.148975 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.149180 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.149181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.149181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.149321 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.149425 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.149609 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.149684 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 17:52:06.151455 6490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:06.153101 6490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:06.160323 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33601 (local address 127.0.0.1:48394)
0928 17:52:06.156638 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.156647 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.156698 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.157237 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.157238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.157240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.157290 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.157292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158155 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.158156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.159078 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.159081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.160014 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.160015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.160077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.160082 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.160182 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.160234 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 17:52:06.160339 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.2:33601)
0928 17:52:06.156979 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.156983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.156993 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:06.157098 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.157100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.157119 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.157239 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.157345 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158063 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.159177 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.159933 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.160015 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.160097 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.160200 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.160238 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260928 17:52:06.161234 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.0.98.1:37301)
0928 17:52:06.156924 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.156930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.157070 (+ 140us) server_negotiation.cc:407] Connection header received
0928 17:52:06.157134 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.157137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.157154 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.157280 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.157471 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158574 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.159795 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.160872 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.160953 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.161020 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.161106 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.161147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260928 17:52:06.161644 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37301 (local address 127.0.0.1:43002)
0928 17:52:06.156991 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.157000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.157068 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.157346 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.157346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.157348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.157394 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.157395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158663 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.158664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.159708 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.159711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.160959 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.160960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.161019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.161024 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.161091 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.161138 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260928 17:52:06.161598 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 (DEFAULT_TABLE table=test-table [id=79da44c235154f618f0b8dc375e5a6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:06.162025 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2b6229c20417cb73f9a3d61ecddd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:06.162432 7396 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 (DEFAULT_TABLE table=test-table [id=79da44c235154f618f0b8dc375e5a6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:06.162757 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2b6229c20417cb73f9a3d61ecddd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:06.164237 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.0.1:53616)
0928 17:52:06.156467 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.156478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.156516 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.157346 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.157346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.157348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.157394 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.157395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158694 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.158695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.159592 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.159594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.160974 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.160974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.161037 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.161040 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.161071 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.161132 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":3}
I20260928 17:52:06.164669 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.156543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.98.3:34983)
0928 17:52:06.157186 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.157192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.157201 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:06.157236 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.157238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.157254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.157466 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.157523 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.158579 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.159686 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.160872 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.164415 (+ 3543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.164504 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.164532 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.164586 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":64,"threads_started":1}
I20260928 17:52:06.165351 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 (DEFAULT_TABLE table=test-table [id=79da44c235154f618f0b8dc375e5a6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:06.165534 7759 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Bootstrap starting.
I20260928 17:52:06.165689 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d2b6229c20417cb73f9a3d61ecddd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:06.165777 7758 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Bootstrap starting.
I20260928 17:52:06.166260 7759 tablet_bootstrap.cc:654] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:06.166455 7758 tablet_bootstrap.cc:654] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:06.166563 7759 log.cc:824] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:06.166806 7758 log.cc:824] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:06.167346 7759 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: No bootstrap required, opened a new log
I20260928 17:52:06.167443 7759 ts_tablet_manager.cc:1397] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:52:06.168699 7758 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: No bootstrap required, opened a new log
I20260928 17:52:06.168769 7758 ts_tablet_manager.cc:1397] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:52:06.169018 7762 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Bootstrap starting.
I20260928 17:52:06.169544 7759 raft_consensus.cc:348] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.169719 7759 raft_consensus.cc:374] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:06.169762 7759 raft_consensus.cc:729] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d175a3b1507455ebd5814f107b8ceeb, State: Initialized, Role: FOLLOWER
I20260928 17:52:06.169687 7762 tablet_bootstrap.cc:654] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:06.169875 7759 consensus_queue.cc:260] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.170136 7759 ts_tablet_manager.cc:1428] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:52:06.170295 7463 heartbeater.cc:503] Master 127.0.98.62:34305 was elected leader, sending a full tablet report...
I20260928 17:52:06.171047 7762 log.cc:824] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:06.171367 7758 raft_consensus.cc:348] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.171510 7758 raft_consensus.cc:374] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:06.171592 7758 raft_consensus.cc:729] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ca743e6a9e45948769e293533d35e2, State: Initialized, Role: FOLLOWER
I20260928 17:52:06.171758 7758 consensus_queue.cc:260] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.171808 7762 tablet_bootstrap.cc:492] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: No bootstrap required, opened a new log
I20260928 17:52:06.171866 7762 ts_tablet_manager.cc:1397] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:52:06.172004 7758 ts_tablet_manager.cc:1428] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:52:06.172369 7600 heartbeater.cc:503] Master 127.0.98.62:34305 was elected leader, sending a full tablet report...
I20260928 17:52:06.174147 7762 raft_consensus.cc:348] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.174268 7762 raft_consensus.cc:374] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:06.174290 7762 raft_consensus.cc:729] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d5ac0fbe78450c9742c5faddb9f213, State: Initialized, Role: FOLLOWER
I20260928 17:52:06.174391 7762 consensus_queue.cc:260] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.174587 7762 ts_tablet_manager.cc:1428] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:52:06.174659 7736 heartbeater.cc:503] Master 127.0.98.62:34305 was elected leader, sending a full tablet report...
W20260928 17:52:06.242722 7602 tablet.cc:2369] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:06.332510 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.329933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39120)
0928 17:52:06.330013 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.330025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.330080 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.330251 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.330251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.330253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.330304 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.330306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.330963 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.330963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.331687 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.331689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.332318 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.332319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.332366 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.332369 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.332415 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.332450 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 17:52:06.332509 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.330008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.98.60:38311)
0928 17:52:06.330067 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.330069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.330081 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:52:06.330135 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.330136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.330158 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.330232 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.330365 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.330883 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.331774 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.332236 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.332302 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.332370 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.332422 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.332449 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:52:06.332954 7293 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 17:52:06.333925 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32b99c0e0ca04cf588be01236a671848. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:06.334031 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: LEARNER
W20260928 17:52:06.352268 7465 tablet.cc:2369] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:52:06.377635 7738 tablet.cc:2369] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:06.406250 7765 raft_consensus.cc:482] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:06.406379 7765 raft_consensus.cc:504] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.406793 7765 leader_election.cc:288] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983), 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:06.409654 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.407045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57143 (local address 127.0.98.1:37301)
0928 17:52:06.407105 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.407107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.407148 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:52:06.407213 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.407214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.407234 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.407302 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.407464 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.407949 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.408936 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.409289 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.409372 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.409381 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.409532 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.409573 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:52:06.409654 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.407107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40655 (local address 127.0.98.3:34983)
0928 17:52:06.407167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.407169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.407213 (+ 44us) server_negotiation.cc:407] Connection header received
0928 17:52:06.407294 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.407295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.407307 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.407374 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:06.407518 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.407986 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.408917 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.409290 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.409369 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.409380 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.409542 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.409580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:52:06.409686 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.406967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37301 (local address 127.0.98.2:57143)
0928 17:52:06.407080 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.407093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.407149 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.407308 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.407308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.407310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.407388 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.407389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.408052 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.408053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.408818 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.408820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.409372 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.409373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.409453 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.409456 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.409527 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.409578 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":4}
I20260928 17:52:06.409701 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.406978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.98.2:40655)
0928 17:52:06.407142 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:06.407154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:06.407229 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:06.407379 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:06.407380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:06.407381 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:06.407449 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.407450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.408071 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.408072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.408824 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:06.408827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.409380 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.409381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:06.409462 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.409465 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:06.409524 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 17:52:06.409573 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":149}
I20260928 17:52:06.410034 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2b6229c20417cb73f9a3d61ecddd8" candidate_uuid: "a3ca743e6a9e45948769e293533d35e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d175a3b1507455ebd5814f107b8ceeb" is_pre_election: true
I20260928 17:52:06.410101 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2b6229c20417cb73f9a3d61ecddd8" candidate_uuid: "a3ca743e6a9e45948769e293533d35e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" is_pre_election: true
I20260928 17:52:06.410210 7689 raft_consensus.cc:2454] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ca743e6a9e45948769e293533d35e2 in term 0.
I20260928 17:52:06.410210 7416 raft_consensus.cc:2454] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ca743e6a9e45948769e293533d35e2 in term 0.
I20260928 17:52:06.410444 7489 leader_election.cc:302] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d175a3b1507455ebd5814f107b8ceeb, a3ca743e6a9e45948769e293533d35e2; no voters:
I20260928 17:52:06.410588 7765 raft_consensus.cc:2781] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:06.410645 7765 raft_consensus.cc:482] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:06.410681 7765 raft_consensus.cc:3037] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:06.411388 7765 raft_consensus.cc:504] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.411543 7765 leader_election.cc:288] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [CANDIDATE]: Term 1 election: Requested vote from peers e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3:34983), 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:06.411707 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2b6229c20417cb73f9a3d61ecddd8" candidate_uuid: "a3ca743e6a9e45948769e293533d35e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d175a3b1507455ebd5814f107b8ceeb"
I20260928 17:52:06.411707 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d2b6229c20417cb73f9a3d61ecddd8" candidate_uuid: "a3ca743e6a9e45948769e293533d35e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d5ac0fbe78450c9742c5faddb9f213"
I20260928 17:52:06.411787 7689 raft_consensus.cc:3037] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:06.411787 7416 raft_consensus.cc:3037] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:06.412700 7416 raft_consensus.cc:2454] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ca743e6a9e45948769e293533d35e2 in term 1.
I20260928 17:52:06.412706 7689 raft_consensus.cc:2454] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ca743e6a9e45948769e293533d35e2 in term 1.
I20260928 17:52:06.412894 7488 leader_election.cc:302] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3ca743e6a9e45948769e293533d35e2, e7d5ac0fbe78450c9742c5faddb9f213; no voters:
I20260928 17:52:06.412993 7765 raft_consensus.cc:2781] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:06.413125 7765 raft_consensus.cc:686] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 1 LEADER]: Becoming Leader. State: Replica: a3ca743e6a9e45948769e293533d35e2, State: Running, Role: LEADER
I20260928 17:52:06.413230 7765 consensus_queue.cc:237] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } }
I20260928 17:52:06.414099 6490 catalog_manager.cc:5495] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ca743e6a9e45948769e293533d35e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a3ca743e6a9e45948769e293533d35e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } health_report { overall_health: HEALTHY } } }
I20260928 17:52:06.414774 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: true } } }
I20260928 17:52:06.414878 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:06.415670 7246 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: attempting to promote NON_VOTER d302a92aad304957a107161e9b64ba98 to VOTER
I20260928 17:52:06.415787 7246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:06.416337 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 17:52:06.416380 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 LEARNER]: Refusing update from remote peer 32b99c0e0ca04cf588be01236a671848: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 17:52:06.416532 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 17:52:06.416622 7774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 17:52:06.417295 7246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d302a92aad304957a107161e9b64ba98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.417451 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d302a92aad304957a107161e9b64ba98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.417492 7292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, d302a92aad304957a107161e9b64ba98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418254 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418346 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:06.418412 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418474 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:06.418428 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418509 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:52:06.418517 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418581 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:06.418464 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32b99c0e0ca04cf588be01236a671848" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:06.418717 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:52:06.447365 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.444424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.2:33601)
0928 17:52:06.444475 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.444477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.444485 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:06.444544 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.444545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.444565 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.444668 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:06.444823 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.445513 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.446579 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.446803 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.446878 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.447042 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:06.447161 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:06.447245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.447270 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.447293 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:52:06.449676 7689 raft_consensus.cc:1261] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Refusing update from remote peer a3ca743e6a9e45948769e293533d35e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:06.449693 7416 raft_consensus.cc:1261] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Refusing update from remote peer a3ca743e6a9e45948769e293533d35e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:06.450063 7772 consensus_queue.cc:1035] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:06.450240 7765 consensus_queue.cc:1035] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:06.453680 7779 mvcc.cc:205] Tried to move back new op lower bound from 7334371026734530560 to 7334371026589110272. Current Snapshot: MvccSnapshot[applied={T|T < 7334371026734530560}]
I20260928 17:52:06.455649 6489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:06.455744 6489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:06.457367 6489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32b99c0e0ca04cf588be01236a671848: Sending DeleteTablet for 3 replicas of tablet 94d2b6229c20417cb73f9a3d61ecddd8
I20260928 17:52:06.457702 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43002
I20260928 17:52:06.457715 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53616
I20260928 17:52:06.458668 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d2b6229c20417cb73f9a3d61ecddd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48394
I20260928 17:52:06.458886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6458
I20260928 17:52:06.464263 7787 tablet_replica.cc:330] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: stopping tablet replica
I20260928 17:52:06.464325 7788 tablet_replica.cc:330] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: stopping tablet replica
I20260928 17:52:06.464422 7787 raft_consensus.cc:2229] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:06.464550 7787 raft_consensus.cc:2258] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:06.465305 7788 raft_consensus.cc:2229] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:06.465379 7787 ts_tablet_manager.cc:1905] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:06.465574 7788 raft_consensus.cc:2258] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:06.466933 7790 tablet_replica.cc:330] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: stopping tablet replica
I20260928 17:52:06.467113 7790 raft_consensus.cc:2229] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:06.467242 7788 ts_tablet_manager.cc:1905] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:06.467278 7790 raft_consensus.cc:2258] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:06.467818 7790 ts_tablet_manager.cc:1905] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:06.469050 7788 ts_tablet_manager.cc:1918] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:06.469133 7788 log.cc:1196] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/94d2b6229c20417cb73f9a3d61ecddd8
I20260928 17:52:06.469434 7788 ts_tablet_manager.cc:1939] T 94d2b6229c20417cb73f9a3d61ecddd8 P e7d5ac0fbe78450c9742c5faddb9f213: Deleting consensus metadata
I20260928 17:52:06.469772 7790 ts_tablet_manager.cc:1918] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 17:52:06.469816 7623 connection.cc:441] server connection from 127.0.0.1:53616 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53616 (request call id 1) could send its response
I20260928 17:52:06.469941 7790 log.cc:1196] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/94d2b6229c20417cb73f9a3d61ecddd8
I20260928 17:52:06.470268 7790 ts_tablet_manager.cc:1939] T 94d2b6229c20417cb73f9a3d61ecddd8 P a3ca743e6a9e45948769e293533d35e2: Deleting consensus metadata
W20260928 17:52:06.470630 7487 connection.cc:441] server connection from 127.0.0.1:48394 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48394 (request call id 1) could send its response
I20260928 17:52:06.470715 7787 ts_tablet_manager.cc:1918] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:06.470799 7787 log.cc:1196] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/94d2b6229c20417cb73f9a3d61ecddd8
I20260928 17:52:06.471091 7787 ts_tablet_manager.cc:1939] T 94d2b6229c20417cb73f9a3d61ecddd8 P 8d175a3b1507455ebd5814f107b8ceeb: Deleting consensus metadata
W20260928 17:52:06.471828 7352 connection.cc:441] server connection from 127.0.0.1:43002 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43002 (request call id 1) could send its response
I20260928 17:52:06.474435 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.458317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.0.98.60:38311)
0928 17:52:06.458356 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.458358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.465845 (+ 7487us) server_negotiation.cc:407] Connection header received
0928 17:52:06.465963 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.465964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.465976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.466019 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.466820 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.467610 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.468797 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.469119 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.469197 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.470906 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.470908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.470908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.470917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.472807 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.473363 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.473364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.473364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.473428 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.473616 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.473616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.473616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.473742 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.473825 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.474259 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.474342 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:52:06.479974 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.471748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.0.98.61:32983)
0928 17:52:06.471792 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.471794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.472230 (+ 436us) server_negotiation.cc:407] Connection header received
0928 17:52:06.472330 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.472331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.472346 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.472377 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.473032 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.473833 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.474897 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.475162 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.475199 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.476968 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.476970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.476970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.476978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.478571 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.479046 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.479047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.479048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.479137 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.479377 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.479378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.479378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.479501 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.479585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.479775 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.479862 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:06.480029 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:06.471425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.0.98.60:38311)
0928 17:52:06.472019 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:06.472021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:06.472030 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:06.472049 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:06.472050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:06.472061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:06.472151 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:06.472842 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.473609 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.474944 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:06.475286 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:06.475355 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:06.477021 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:06.477023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:06.477023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:06.477029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:06.478595 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.479077 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.479078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.479078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.479134 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:06.479343 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:06.479344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:06.479344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:06.479492 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:06.479597 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:06.479702 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:52:06.479804 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":83,"threads_started":1}
I20260928 17:52:07.454593 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.454357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:57641)
0928 17:52:07.454476 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.454551 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
W20260928 17:52:07.454815 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34305 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.455216 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.454967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:59479)
0928 17:52:07.455113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.455181 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260928 17:52:07.455415 7600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34305 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.456208 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.455999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:52119)
0928 17:52:07.456105 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.456171 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":47,"threads_started":1}
W20260928 17:52:07.456400 7463 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34305 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.957628 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32b99c0e0ca04cf588be01236a671848)
I20260928 17:52:07.957780 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:07.957935 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32b99c0e0ca04cf588be01236a671848)
I20260928 17:52:07.958135 7801 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311)
I20260928 17:52:07.958081 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:07.958495 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39546)
0928 17:52:07.958399 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.958452 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
W20260928 17:52:07.958671 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.958690 7802 leader_election.cc:288] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), ba4a662729254e25b0c2de598f47685f (127.0.98.61:32983)
I20260928 17:52:07.959074 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39558)
0928 17:52:07.958979 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.959025 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":46,"threads_started":1}
I20260928 17:52:07.959074 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39574)
0928 17:52:07.959012 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.959038 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 17:52:07.959143 7078 leader_election.cc:334] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
W20260928 17:52:07.959261 7267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.959686 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.959584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39584)
0928 17:52:07.959629 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.959660 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:52:07.959939 7267 leader_election.cc:334] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.961782 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32983 (local address 127.0.0.1:53088)
0928 17:52:07.958816 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.958827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:07.958900 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:07.959094 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:07.959094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:07.959097 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:07.959145 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:07.959147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.960035 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.960036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:07.960938 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:07.960940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.961490 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:07.961555 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:07.961559 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:07.961627 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:52:07.961675 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":40,"threads_started":1}
I20260928 17:52:07.961771 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.0.98.61:32983)
0928 17:52:07.958784 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:07.958786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:07.958875 (+ 89us) server_negotiation.cc:407] Connection header received
0928 17:52:07.958976 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:07.958977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:07.958987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:07.959026 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:07.959228 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.959961 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961030 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.961399 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961488 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:07.961499 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:07.961632 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 17:52:07.961679 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":44,"threads_started":1}
I20260928 17:52:07.961853 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38311 (local address 127.0.0.1:39136)
0928 17:52:07.958635 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.958646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:07.958731 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:07.958955 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:07.958957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:07.958959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:07.959018 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:07.959020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.959939 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.959940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:07.960956 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:07.960959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.961518 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:07.961581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:07.961585 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:07.961655 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:52:07.961700 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":72,"threads_started":1}
I20260928 17:52:07.961791 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.958566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.98.60:38311)
0928 17:52:07.958646 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:07.958648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:07.958767 (+ 119us) server_negotiation.cc:407] Connection header received
0928 17:52:07.958809 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:07.958810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:07.958823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:07.958901 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:07.959099 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.959848 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961060 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:07.961428 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:07.961516 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:07.961528 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:07.961659 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 17:52:07.961700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63,"thread_start_us":33,"threads_started":1}
I20260928 17:52:07.962112 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d302a92aad304957a107161e9b64ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ba4a662729254e25b0c2de598f47685f" is_pre_election: true
I20260928 17:52:07.962188 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a662729254e25b0c2de598f47685f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d302a92aad304957a107161e9b64ba98" is_pre_election: true
I20260928 17:52:07.962220 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d302a92aad304957a107161e9b64ba98 in term 1.
I20260928 17:52:07.962278 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4a662729254e25b0c2de598f47685f in term 1.
I20260928 17:52:07.962364 7270 leader_election.cc:302] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba4a662729254e25b0c2de598f47685f, d302a92aad304957a107161e9b64ba98; no voters: 32b99c0e0ca04cf588be01236a671848
I20260928 17:52:07.962414 7079 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba4a662729254e25b0c2de598f47685f, d302a92aad304957a107161e9b64ba98; no voters: 32b99c0e0ca04cf588be01236a671848
I20260928 17:52:07.962452 7802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 17:52:07.962488 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32b99c0e0ca04cf588be01236a671848)
I20260928 17:52:07.962500 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 17:52:07.962517 7802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:52:07.962544 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32b99c0e0ca04cf588be01236a671848)
I20260928 17:52:07.962574 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:52:07.963382 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:07.963382 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:07.963524 7801 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 election: Requested vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311)
I20260928 17:52:07.963522 7802 leader_election.cc:288] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 election: Requested vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), ba4a662729254e25b0c2de598f47685f (127.0.98.61:32983)
I20260928 17:52:07.963696 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a662729254e25b0c2de598f47685f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d302a92aad304957a107161e9b64ba98"
I20260928 17:52:07.963740 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d302a92aad304957a107161e9b64ba98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ba4a662729254e25b0c2de598f47685f"
I20260928 17:52:07.963806 7104 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d302a92aad304957a107161e9b64ba98 in current term 2: Already voted for candidate ba4a662729254e25b0c2de598f47685f in this term.
I20260928 17:52:07.963757 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.963625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39614)
0928 17:52:07.963699 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.963729 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 17:52:07.963795 7292 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ba4a662729254e25b0c2de598f47685f in current term 2: Already voted for candidate d302a92aad304957a107161e9b64ba98 in this term.
I20260928 17:52:07.964143 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.964010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39644)
0928 17:52:07.964075 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.964120 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 17:52:07.964231 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.963750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39630)
0928 17:52:07.964157 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.964187 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260928 17:52:07.964231 7267 leader_election.cc:334] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.964272 7267 leader_election.cc:302] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d302a92aad304957a107161e9b64ba98; no voters: 32b99c0e0ca04cf588be01236a671848, ba4a662729254e25b0c2de598f47685f
I20260928 17:52:07.964327 7802 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 17:52:07.964520 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:07.964446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39656)
0928 17:52:07.964484 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:07.964506 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 17:52:07.964581 7078 leader_election.cc:334] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:07.964632 7078 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba4a662729254e25b0c2de598f47685f; no voters: 32b99c0e0ca04cf588be01236a671848, d302a92aad304957a107161e9b64ba98
I20260928 17:52:07.964697 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 17:52:08.455422 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:08.455157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:45003)
0928 17:52:08.455295 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:08.455354 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260928 17:52:08.455914 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:08.455725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:45987)
0928 17:52:08.455836 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:08.455871 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260928 17:52:08.456893 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:08.456713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:40397)
0928 17:52:08.456822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:08.456852 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260928 17:52:09.456231 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.455873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:35501)
0928 17:52:09.456103 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.456163 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260928 17:52:09.456598 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.456400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:49867)
0928 17:52:09.456504 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.456540 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260928 17:52:09.457386 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.457248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:43399)
0928 17:52:09.457330 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.457353 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":40,"threads_started":1}
I20260928 17:52:09.785599 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:09.785696 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:09.785852 7817 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311)
I20260928 17:52:09.786125 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a662729254e25b0c2de598f47685f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d302a92aad304957a107161e9b64ba98" is_pre_election: true
I20260928 17:52:09.786221 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4a662729254e25b0c2de598f47685f in term 2.
I20260928 17:52:09.786240 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.786028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39868)
0928 17:52:09.786147 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.786179 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260928 17:52:09.786403 7079 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4a662729254e25b0c2de598f47685f, d302a92aad304957a107161e9b64ba98; no voters:
I20260928 17:52:09.786501 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 17:52:09.786542 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:09.786568 7817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:52:09.786767 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.786674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39878)
0928 17:52:09.786718 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.786747 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:52:09.786861 7078 leader_election.cc:334] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:09.798331 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:09.798727 7817 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 election: Requested vote from peers 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305), d302a92aad304957a107161e9b64ba98 (127.0.98.60:38311)
I20260928 17:52:09.798849 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.798613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39888)
0928 17:52:09.798674 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.798716 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 17:52:09.798910 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a662729254e25b0c2de598f47685f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d302a92aad304957a107161e9b64ba98"
I20260928 17:52:09.798979 7292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:52:09.799578 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba4a662729254e25b0c2de598f47685f in term 3.
I20260928 17:52:09.799793 7079 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4a662729254e25b0c2de598f47685f, d302a92aad304957a107161e9b64ba98; no voters:
I20260928 17:52:09.799958 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.799861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39898)
0928 17:52:09.799907 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.799937 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:52:09.800036 7078 leader_election.cc:334] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
I20260928 17:52:09.800148 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 3 FOLLOWER]: Leader election won for term 3
I20260928 17:52:09.800302 7817 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [term 3 LEADER]: Becoming Leader. State: Replica: ba4a662729254e25b0c2de598f47685f, State: Running, Role: LEADER
I20260928 17:52:09.800393 7817 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } }
I20260928 17:52:09.800757 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4a662729254e25b0c2de598f47685f. Latest consensus state: current_term: 3 leader_uuid: "ba4a662729254e25b0c2de598f47685f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:09.800879 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:09.801100 7822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:09.801599 7822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:09.801658 7822 catalog_manager.cc:1259] Loaded cluster ID: a0ca96e2187a4515b09ef65b3ac703c5
I20260928 17:52:09.801699 7822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:09.801923 7822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:09.802019 7822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Loaded TSK: 0
I20260928 17:52:09.802182 7822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:52:09.888236 7093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:09.889931 7093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:09.891027 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.890914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39920)
0928 17:52:09.890965 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.891000 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:52:09.891057 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [term 3 FOLLOWER]: Refusing update from remote peer ba4a662729254e25b0c2de598f47685f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 17:52:09.891390 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.891297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.0.1:39932)
0928 17:52:09.891333 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.891371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 17:52:09.891335 7820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 17:52:09.891490 7078 consensus_peers.cc:487] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f -> Peer 32b99c0e0ca04cf588be01236a671848 (127.0.98.62:34305): Couldn't send request to peer 32b99c0e0ca04cf588be01236a671848. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 17:52:09.891980 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4a662729254e25b0c2de598f47685f. Latest consensus state: current_term: 3 leader_uuid: "ba4a662729254e25b0c2de598f47685f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:09.892131 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:52:09.892452 7823 mvcc.cc:205] Tried to move back new op lower bound from 7334371040831660032 to 7334371040463142912. Current Snapshot: MvccSnapshot[applied={T|T < 7334371040831660032}]
I20260928 17:52:09.892601 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ba4a662729254e25b0c2de598f47685f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:09.892693 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:09.892759 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ba4a662729254e25b0c2de598f47685f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "32b99c0e0ca04cf588be01236a671848" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34305 } } peers { permanent_uuid: "ba4a662729254e25b0c2de598f47685f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "d302a92aad304957a107161e9b64ba98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38311 } attrs { promote: false } } }
I20260928 17:52:09.892856 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P d302a92aad304957a107161e9b64ba98 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:52:09.898020 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37301 (local address 127.0.0.1:43010)
0928 17:52:09.894727 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.894737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.894786 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.895027 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:09.895028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:09.895030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:09.895104 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.895105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.895747 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.895748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.897045 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.897047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.897812 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.897869 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.897873 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:09.897915 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 17:52:09.897949 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260928 17:52:09.898078 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.2:33601)
0928 17:52:09.894843 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.894846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.894853 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:09.894880 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.894881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.894891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.895240 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:09.895283 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.896141 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897294 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.897755 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897824 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.897835 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.897972 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.898005 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":51,"threads_started":1}
I20260928 17:52:09.898106 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33601 (local address 127.0.0.1:48416)
0928 17:52:09.894694 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.894706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.894757 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.895125 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:09.895126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:09.895128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:09.895176 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.895177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.896230 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.896231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.897217 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.897219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.897825 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.897885 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.897889 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:09.897965 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:52:09.898005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260928 17:52:09.898598 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 5079343dc07548dcb549c6ad73072ed5 (DEFAULT_TABLE table=test-table [id=1c8da363d28a4e27b2ad3d490b8b1b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:09.898720 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.98.1:37301)
0928 17:52:09.894874 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.894876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.894884 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:09.894911 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.894912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.894922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.894978 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:09.895182 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.895668 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897130 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.897703 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.897771 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.897782 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.898574 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.898624 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260928 17:52:09.898742 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5079343dc07548dcb549c6ad73072ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:09.898967 7396 tablet_service.cc:1467] Processing CreateTablet for tablet 5079343dc07548dcb549c6ad73072ed5 (DEFAULT_TABLE table=test-table [id=1c8da363d28a4e27b2ad3d490b8b1b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:09.899086 7396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5079343dc07548dcb549c6ad73072ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:09.899242 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.0.1:53630)
0928 17:52:09.894981 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.894989 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.895037 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.895223 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:09.895224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:09.895226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:09.895263 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.895264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.896991 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.896992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.898100 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.898102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.898960 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.898961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.899022 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.899027 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:09.899116 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:52:09.899155 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":90,"threads_started":1}
I20260928 17:52:09.899650 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.894633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.98.3:34983)
0928 17:52:09.894724 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.894726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.895032 (+ 306us) server_negotiation.cc:407] Connection header received
0928 17:52:09.895091 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.895092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.895104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.895178 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:09.896172 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.896858 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.898219 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.898763 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.898955 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.898971 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.899506 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.899567 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":43,"threads_started":1}
I20260928 17:52:09.900326 7836 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: Bootstrap starting.
I20260928 17:52:09.900465 7835 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: Bootstrap starting.
I20260928 17:52:09.900570 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 5079343dc07548dcb549c6ad73072ed5 (DEFAULT_TABLE table=test-table [id=1c8da363d28a4e27b2ad3d490b8b1b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:09.900687 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5079343dc07548dcb549c6ad73072ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:09.900719 7836 tablet_bootstrap.cc:654] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:09.900892 7835 tablet_bootstrap.cc:654] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:09.901346 7836 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: No bootstrap required, opened a new log
I20260928 17:52:09.901404 7836 ts_tablet_manager.cc:1397] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:09.901422 7835 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: No bootstrap required, opened a new log
I20260928 17:52:09.901463 7835 ts_tablet_manager.cc:1397] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:09.901526 7836 raft_consensus.cc:348] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.901595 7836 raft_consensus.cc:374] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:09.901620 7836 raft_consensus.cc:729] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ca743e6a9e45948769e293533d35e2, State: Initialized, Role: FOLLOWER
I20260928 17:52:09.901599 7835 raft_consensus.cc:348] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.901665 7835 raft_consensus.cc:374] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:09.901688 7835 raft_consensus.cc:729] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d175a3b1507455ebd5814f107b8ceeb, State: Initialized, Role: FOLLOWER
I20260928 17:52:09.901678 7836 consensus_queue.cc:260] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.901753 7835 consensus_queue.cc:260] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.901870 7836 ts_tablet_manager.cc:1428] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:09.901948 7835 ts_tablet_manager.cc:1428] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:09.902354 7841 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: Bootstrap starting.
I20260928 17:52:09.902381 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.902286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:58149)
0928 17:52:09.902322 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.902351 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 17:52:09.902586 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.902316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:57077)
0928 17:52:09.902527 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.902558 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260928 17:52:09.902750 7841 tablet_bootstrap.cc:654] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:09.903234 7464 heartbeater.cc:503] Master 127.0.98.61:32983 was elected leader, sending a full tablet report...
I20260928 17:52:09.903358 7841 tablet_bootstrap.cc:492] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: No bootstrap required, opened a new log
I20260928 17:52:09.903415 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.903317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:47011)
0928 17:52:09.903361 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.903386 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:52:09.903399 7841 ts_tablet_manager.cc:1397] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:09.903671 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.903572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:59087)
0928 17:52:09.903612 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.903647 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:52:09.903749 7841 raft_consensus.cc:348] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.903827 7841 raft_consensus.cc:374] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:09.903837 7601 heartbeater.cc:503] Master 127.0.98.61:32983 was elected leader, sending a full tablet report...
I20260928 17:52:09.903853 7841 raft_consensus.cc:729] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d5ac0fbe78450c9742c5faddb9f213, State: Initialized, Role: FOLLOWER
I20260928 17:52:09.903918 7841 consensus_queue.cc:260] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.904150 7841 ts_tablet_manager.cc:1428] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:09.906052 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.904366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:40127)
0928 17:52:09.904402 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.904432 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 17:52:09.906247 7737 heartbeater.cc:503] Master 127.0.98.61:32983 was elected leader, sending a full tablet report...
I20260928 17:52:09.906548 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.906452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:57123)
0928 17:52:09.906485 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.906517 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 17:52:09.924341 7843 raft_consensus.cc:482] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:09.924441 7843 raft_consensus.cc:504] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.924830 7843 leader_election.cc:288] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601), 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:09.927845 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.925010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37301 (local address 127.0.98.3:57419)
0928 17:52:09.925087 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.925099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.925173 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.925364 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:09.925365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:09.925368 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:09.925450 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.925453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.926164 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.926165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.927028 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.927031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.927570 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.927650 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.927664 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:52:09.927716 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:52:09.927749 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":56}
I20260928 17:52:09.927863 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.925094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57419 (local address 127.0.98.1:37301)
0928 17:52:09.925144 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.925146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.925157 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:09.925246 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.925247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.925260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.925337 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:09.925526 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.926069 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927137 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.927496 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927551 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.927561 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.927748 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.927796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:52:09.927930 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.925102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58843 (local address 127.0.98.2:33601)
0928 17:52:09.925157 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.925159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.925358 (+ 199us) server_negotiation.cc:407] Connection header received
0928 17:52:09.925396 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.925397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.925412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.925472 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:09.925668 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.926129 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927124 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.927536 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927609 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.927620 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.927818 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.927860 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:52:09.928213 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.925045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33601 (local address 127.0.98.3:58843)
0928 17:52:09.925277 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.925288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.925348 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.925480 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:09.925481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:09.925496 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:09.925578 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.925580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.926212 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.926213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.927028 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:09.927030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.927643 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.927644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:09.927733 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.927738 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:09.927795 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:52:09.927836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":51,"thread_start_us":116,"threads_started":1}
I20260928 17:52:09.928238 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5079343dc07548dcb549c6ad73072ed5" candidate_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d175a3b1507455ebd5814f107b8ceeb" is_pre_election: true
I20260928 17:52:09.928344 7416 raft_consensus.cc:2454] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d5ac0fbe78450c9742c5faddb9f213 in term 0.
I20260928 17:52:09.928531 7625 leader_election.cc:302] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d175a3b1507455ebd5814f107b8ceeb, e7d5ac0fbe78450c9742c5faddb9f213; no voters:
I20260928 17:52:09.928514 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5079343dc07548dcb549c6ad73072ed5" candidate_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ca743e6a9e45948769e293533d35e2" is_pre_election: true
I20260928 17:52:09.928646 7553 raft_consensus.cc:2454] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7d5ac0fbe78450c9742c5faddb9f213 in term 0.
I20260928 17:52:09.928669 7843 raft_consensus.cc:2781] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:09.928699 7843 raft_consensus.cc:482] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:09.928722 7843 raft_consensus.cc:3037] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:09.929409 7843 raft_consensus.cc:504] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.929569 7843 leader_election.cc:288] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [CANDIDATE]: Term 1 election: Requested vote from peers a3ca743e6a9e45948769e293533d35e2 (127.0.98.2:33601), 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301)
I20260928 17:52:09.929746 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5079343dc07548dcb549c6ad73072ed5" candidate_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d175a3b1507455ebd5814f107b8ceeb"
I20260928 17:52:09.929833 7416 raft_consensus.cc:3037] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:09.929785 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5079343dc07548dcb549c6ad73072ed5" candidate_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3ca743e6a9e45948769e293533d35e2"
I20260928 17:52:09.929867 7553 raft_consensus.cc:3037] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:09.930573 7416 raft_consensus.cc:2454] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d5ac0fbe78450c9742c5faddb9f213 in term 1.
I20260928 17:52:09.930631 7553 raft_consensus.cc:2454] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7d5ac0fbe78450c9742c5faddb9f213 in term 1.
I20260928 17:52:09.930742 7625 leader_election.cc:302] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d175a3b1507455ebd5814f107b8ceeb, e7d5ac0fbe78450c9742c5faddb9f213; no voters:
I20260928 17:52:09.930837 7843 raft_consensus.cc:2781] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:09.930968 7843 raft_consensus.cc:686] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 LEADER]: Becoming Leader. State: Replica: e7d5ac0fbe78450c9742c5faddb9f213, State: Running, Role: LEADER
I20260928 17:52:09.931041 7843 consensus_queue.cc:237] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } }
I20260928 17:52:09.931414 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.931291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:38385)
0928 17:52:09.931350 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.931388 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:52:09.931824 7093 catalog_manager.cc:5495] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7d5ac0fbe78450c9742c5faddb9f213 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7d5ac0fbe78450c9742c5faddb9f213" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34983 } health_report { overall_health: HEALTHY } } }
I20260928 17:52:09.943714 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.941359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.0.98.3:34983)
0928 17:52:09.941412 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:09.941413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:09.941422 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:09.941497 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:09.941498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:09.941518 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:09.941601 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:09.941743 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.942221 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.942957 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:09.943174 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:09.943244 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:09.943401 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:09.943500 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:09.943576 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:09.943618 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:52:09.943639 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:52:09.944746 7416 raft_consensus.cc:1261] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb [term 1 FOLLOWER]: Refusing update from remote peer e7d5ac0fbe78450c9742c5faddb9f213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:09.944932 7553 raft_consensus.cc:1261] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Refusing update from remote peer e7d5ac0fbe78450c9742c5faddb9f213: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:09.944988 7845 consensus_queue.cc:1035] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d175a3b1507455ebd5814f107b8ceeb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:09.945199 7843 consensus_queue.cc:1035] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3ca743e6a9e45948769e293533d35e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:09.945649 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.945546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:38299)
0928 17:52:09.945586 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.945613 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:52:09.946291 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.946145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:41191)
0928 17:52:09.946233 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.946261 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 17:52:09.946708 7849 mvcc.cc:205] Tried to move back new op lower bound from 7334371041051709440 to 7334371040998064128. Current Snapshot: MvccSnapshot[applied={T|T < 7334371041051709440}]
I20260928 17:52:09.946866 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.946778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:39213)
0928 17:52:09.946816 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.946840 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:52:09.947139 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.947037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.1:49213)
0928 17:52:09.947074 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.947099 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:52:09.947322 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.947241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.3:56411)
0928 17:52:09.947272 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.947294 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 17:52:09.947881 7851 mvcc.cc:205] Tried to move back new op lower bound from 7334371041051709440 to 7334371040998064128. Current Snapshot: MvccSnapshot[applied={T|T < 7334371041051709440 or (T in {7334371041051709440})}]
I20260928 17:52:09.948077 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.947941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34305 (local address 127.0.98.2:48041)
0928 17:52:09.948023 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.948049 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 17:52:09.949754 7093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:09.949849 7093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:09.951274 7093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba4a662729254e25b0c2de598f47685f: Sending DeleteTablet for 3 replicas of tablet 5079343dc07548dcb549c6ad73072ed5
I20260928 17:52:09.951594 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 5079343dc07548dcb549c6ad73072ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:09 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48416
I20260928 17:52:09.951625 7396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5079343dc07548dcb549c6ad73072ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:09 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43010
I20260928 17:52:09.951628 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 5079343dc07548dcb549c6ad73072ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:09 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53630
I20260928 17:52:09.951790 7857 tablet_replica.cc:330] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2: stopping tablet replica
I20260928 17:52:09.951838 7857 raft_consensus.cc:2229] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:09.951880 7857 raft_consensus.cc:2258] T 5079343dc07548dcb549c6ad73072ed5 P a3ca743e6a9e45948769e293533d35e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:09.951983 7858 tablet_replica.cc:330] T 5079343dc07548dcb549c6ad73072ed5 P 8d175a3b1507455ebd5814f107b8ceeb: stopping tablet replica
I20260928 17:52:09.952183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7242
I20260928 17:52:09.952137 7859 tablet_replica.cc:330] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: stopping tablet replica
I20260928 17:52:09.952982 7859 raft_consensus.cc:2229] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:09.953167 7859 raft_consensus.cc:2258] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:09.953522 7859 ts_tablet_manager.cc:1905] T 5079343dc07548dcb549c6ad73072ed5 P e7d5ac0fbe78450c9742c5faddb9f213: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 17:52:09.958863 7832 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:09.958133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37301 (local address 127.0.0.1:43014)
0928 17:52:09.958176 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.958188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:09.958225 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:09.958818 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 17:52:09.959074 7081 catalog_manager.cc:4584] TS 8d175a3b1507455ebd5814f107b8ceeb (127.0.98.1:37301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5079343dc07548dcb549c6ad73072ed5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:52:09.959120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7470
I20260928 17:52:09.965227 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.965098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33601 (local address 127.0.0.1:48424)
0928 17:52:09.965154 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.965182 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:52:09.965323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7606
I20260928 17:52:09.975718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 6954
I20260928 17:52:09.975736 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:09.975598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34983 (local address 127.0.0.1:53640)
0928 17:52:09.975667 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:09.975694 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:34983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:52:09.980659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7148
2026-09-28T17:52:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6556 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:10 dist-test-slave-zqlt krb5kdc[7862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:10 dist-test-slave-zqlt krb5kdc[7862](info): set up 2 sockets
Sep 28 17:52:10 dist-test-slave-zqlt krb5kdc[7862](info): commencing operation
krb5kdc: starting...
W20260928 17:52:11.024719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 17:52:11.141157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:11.222647 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:11.222918 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:11.222951 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:11.224215 7875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:11.224270 7875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 17:52:11.224291 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:11.224314 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:11.224342 7875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:11.224371 7875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:11.225855 7875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34365
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:11.226138 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:11.226368 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:11.231233 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:11.231339 7875 server_base.cc:1034] running on GCE node
W20260928 17:52:11.231431 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.231276 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:11.231814 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:11.232219 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:11.233407 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790617931233394 us; error 31 us; skew 500 ppm
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:11.236153 7875 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 17:52:11.236639 7875 webserver.cc:460] Webserver started at http://127.0.98.62:34065/ using document root <none> and password file <none>
I20260928 17:52:11.236871 7875 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:11.236935 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:11.237069 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:11.238190 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "5b70744b1247412b98cf8b62214166c5"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "1e93bdd5934121117cec4da63ca6ecf5"
server_key_iv: "9444dbde311e9d812165485840133024"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.238579 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b70744b1247412b98cf8b62214166c5"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "1e93bdd5934121117cec4da63ca6ecf5"
server_key_iv: "9444dbde311e9d812165485840133024"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.239969 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:11.240748 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.240911 7875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:11.241014 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "5b70744b1247412b98cf8b62214166c5"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "1e93bdd5934121117cec4da63ca6ecf5"
server_key_iv: "9444dbde311e9d812165485840133024"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.241086 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:11.256539 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:11.258495 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:11.258661 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:11.263072 7875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34365
I20260928 17:52:11.263176 7952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34365 every 8 connection(s)
I20260928 17:52:11.263474 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:11.264206 7953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.265967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7875
I20260928 17:52:11.266074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:11.266258 392 external_mini_cluster.cc:1428] Setting key 34b997ffb96b0b3b56c6678c168cc6df
I20260928 17:52:11.266552 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Bootstrap starting.
I20260928 17:52:11.267282 7953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.267606 7953 log.cc:824] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:11.268383 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: No bootstrap required, opened a new log
I20260928 17:52:11.270180 7953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } }
I20260928 17:52:11.270314 7953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.270362 7953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b70744b1247412b98cf8b62214166c5, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.270490 7953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } }
I20260928 17:52:11.270566 7953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:11.270588 7953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:11.270654 7953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:11.271210 7953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } }
I20260928 17:52:11.271323 7953 leader_election.cc:302] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b70744b1247412b98cf8b62214166c5; no voters:
I20260928 17:52:11.271492 7953 leader_election.cc:288] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:11.271549 7958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:11.271750 7958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b70744b1247412b98cf8b62214166c5, State: Running, Role: LEADER
I20260928 17:52:11.271770 7953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:11.271847 7958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } }
I20260928 17:52:11.272238 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b70744b1247412b98cf8b62214166c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } } }
I20260928 17:52:11.272325 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:11.272500 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b70744b1247412b98cf8b62214166c5. Latest consensus state: current_term: 1 leader_uuid: "5b70744b1247412b98cf8b62214166c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b70744b1247412b98cf8b62214166c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34365 } } }
I20260928 17:52:11.272571 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:11.272887 7966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:11.273388 7966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:11.275256 7966 catalog_manager.cc:1347] Generated new cluster ID: 1f496034f9974b5392493dd2cc35b994
I20260928 17:52:11.275298 7966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:11.278755 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.266786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.0.98.62:34365)
0928 17:52:11.266975 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.266987 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.267021 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:52:11.267673 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.267683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.267746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.267893 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.268632 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.269519 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.270290 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.270879 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.270995 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.274625 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.274629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.274629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.274641 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.276788 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.277190 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.277193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.277194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.277314 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.277524 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.277525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.277525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.277855 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.278009 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.278129 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.278286 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260928 17:52:11.292268 7966 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:11.292804 7966 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:11.301546 7966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Generated new TSK 0
I20260928 17:52:11.301733 7966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 17:52:11.331218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34365
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:11.414680 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:11.414924 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:11.414975 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:11.416180 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:11.416241 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:11.416275 7979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:52:11.416314 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:11.417784 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:11.418047 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:11.418273 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:11.423175 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.423175 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.423166 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:11.423166 7979 server_base.cc:1034] running on GCE node
I20260928 17:52:11.423528 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:11.423803 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:11.424966 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790617931424945 us; error 35 us; skew 500 ppm
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:11.427747 7979 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 17:52:11.428318 7979 webserver.cc:460] Webserver started at http://127.0.98.1:41397/ using document root <none> and password file <none>
I20260928 17:52:11.428550 7979 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:11.428614 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:11.428747 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:11.429831 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb66ff1dd84247758e80314c6cd204ed"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "05c4df9b7b67d5a765748ce940acaa1f"
server_key_iv: "2d45b6a8267977ea1ca114334d3565a0"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.430200 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb66ff1dd84247758e80314c6cd204ed"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "05c4df9b7b67d5a765748ce940acaa1f"
server_key_iv: "2d45b6a8267977ea1ca114334d3565a0"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.431507 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:11.432268 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.432417 7979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:11.432518 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "eb66ff1dd84247758e80314c6cd204ed"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "05c4df9b7b67d5a765748ce940acaa1f"
server_key_iv: "2d45b6a8267977ea1ca114334d3565a0"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.432600 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:11.462138 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:11.464440 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:11.464622 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:11.465282 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:11.465334 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.465372 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:11.465395 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:11.472985 7979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46139
I20260928 17:52:11.473100 8109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46139 every 8 connection(s)
I20260928 17:52:11.473434 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:11.475612 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.465826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54559 (local address 127.0.98.62:34365)
0928 17:52:11.465895 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.465898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.466177 (+ 279us) server_negotiation.cc:407] Connection header received
0928 17:52:11.466939 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.466941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.466956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.467016 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.468029 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.468602 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.469667 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.469891 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.469974 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.472585 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.472586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.472587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.472595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.474286 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.474718 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.474719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.474720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.474775 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.475004 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.475006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.475006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.475151 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.475223 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.475460 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.475544 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:52:11.475806 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.465823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.1:54559)
0928 17:52:11.466009 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.466060 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.466818 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.467111 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.467115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.467559 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.467906 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.467914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.468706 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.468707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.469560 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.469563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.469994 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.469995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.470116 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.470689 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.470698 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.472490 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.474379 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.474385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.474387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.474644 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.474820 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.474820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.474820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.474909 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.475220 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.475224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.475345 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.475429 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.475556 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":118,"threads_started":1}
I20260928 17:52:11.476411 8110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34365
I20260928 17:52:11.476491 8110 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:11.476687 8110 heartbeater.cc:511] Master 127.0.98.62:34365 requested a full tablet report, sending...
I20260928 17:52:11.476806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 7979
I20260928 17:52:11.476931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:11.477164 392 external_mini_cluster.cc:1428] Setting key 2feef5b1514dff8d4f5ea6c36a868035
I20260928 17:52:11.477263 7906 ts_manager.cc:194] Registered new tserver with Master: eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139)
I20260928 17:52:11.478135 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54559
I20260928 17:52:11.483952 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.477581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.1:37619)
0928 17:52:11.477713 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.477724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.477793 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.477924 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.477925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.478175 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.478262 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.478264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.478916 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.478917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.479863 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.479866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.480262 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.480263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.480343 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.480831 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.480835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.481138 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.482924 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.482925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.482925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.483100 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.483316 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.483316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.483317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.483369 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.483683 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.483684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.483724 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.483804 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.483884 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
I20260928 17:52:11.484023 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.477531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37619 (local address 127.0.98.62:34365)
0928 17:52:11.477572 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.477574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.477776 (+ 202us) server_negotiation.cc:407] Connection header received
0928 17:52:11.477859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.477859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.477869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.477918 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.478341 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.478807 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.479952 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.480189 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.480248 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.481221 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.481222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.481223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.481230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.482830 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.483182 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.483183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.483183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.483245 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.483439 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.483440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.483440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.483585 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.483671 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.483850 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.483925 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 17:52:11.502691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34365
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:11.585490 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:11.585695 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:11.585718 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:11.586853 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:11.586917 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:11.586964 8119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:52:11.587011 8119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:11.588498 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:11.588747 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:11.588975 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:11.593855 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.593856 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.593855 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:11.594079 8119 server_base.cc:1034] running on GCE node
I20260928 17:52:11.594290 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:11.594530 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:11.595705 8119 hybrid_clock.cc:648] HybridClock initialized: now 1790617931595683 us; error 37 us; skew 500 ppm
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:11.598184 8119 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 17:52:11.598608 8119 webserver.cc:460] Webserver started at http://127.0.98.2:39775/ using document root <none> and password file <none>
I20260928 17:52:11.598784 8119 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:11.598829 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:11.598920 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:11.599898 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa41f7c541354afe9ba9f923e87c2c34"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "a7c0d11782f64367404cb62d2d02422a"
server_key_iv: "d329eb533744b390adca0263b856574f"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.600385 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa41f7c541354afe9ba9f923e87c2c34"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "a7c0d11782f64367404cb62d2d02422a"
server_key_iv: "d329eb533744b390adca0263b856574f"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.601637 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:11.602312 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.602464 8119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:11.602561 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "fa41f7c541354afe9ba9f923e87c2c34"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "a7c0d11782f64367404cb62d2d02422a"
server_key_iv: "d329eb533744b390adca0263b856574f"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.602631 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:11.620448 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:11.622799 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:11.622959 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:11.623536 8119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:11.623593 8119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.623631 8119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:11.623667 8119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:11.631109 8119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34001
I20260928 17:52:11.631189 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34001 every 8 connection(s)
I20260928 17:52:11.631526 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:11.633869 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.623962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42339 (local address 127.0.98.62:34365)
0928 17:52:11.624031 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.624033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.624310 (+ 277us) server_negotiation.cc:407] Connection header received
0928 17:52:11.625087 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.625088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.625105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.625153 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.626099 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.626626 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.628339 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.628574 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.628666 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.631247 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.631249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.631249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.631258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.632803 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.633160 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.633161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.633162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.633211 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.633380 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.633381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.633381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.633475 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.633522 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.633748 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.633808 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:52:11.634032 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.623948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.2:42339)
0928 17:52:11.624149 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.624203 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.624949 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.625240 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.625244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.625670 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.625983 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.625992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.626702 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.626703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.628247 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.628251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.628684 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.628685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.628855 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.629395 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.629406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.631157 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.632871 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.632875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.632876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.633095 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.633251 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.633251 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.633252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.633335 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.633536 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.633540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.633647 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.633727 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.633841 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260928 17:52:11.634521 8250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34365
I20260928 17:52:11.634603 8250 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:11.634794 8250 heartbeater.cc:511] Master 127.0.98.62:34365 requested a full tablet report, sending...
I20260928 17:52:11.635205 7907 ts_manager.cc:194] Registered new tserver with Master: fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001)
I20260928 17:52:11.635660 7907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42339
I20260928 17:52:11.637639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8119
I20260928 17:52:11.637763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:11.637966 392 external_mini_cluster.cc:1428] Setting key 8deafb3da8dc694d6a669c0707286800
I20260928 17:52:11.641592 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.635950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49855 (local address 127.0.98.62:34365)
0928 17:52:11.635991 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.635993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.636105 (+ 112us) server_negotiation.cc:407] Connection header received
0928 17:52:11.636182 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.636183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.636193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.636237 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.636608 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.637127 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.637825 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.638054 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.638115 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.639003 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.639005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.639005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.639012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.640442 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.640792 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.640793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.640794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.640850 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.641047 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.641048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.641048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.641189 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.641263 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.641424 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.641507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:52:11.641599 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.635890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.2:49855)
0928 17:52:11.636024 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.636034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.636124 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.636250 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.636251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.636459 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.636534 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.636536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.637219 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.637220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.637760 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.637762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.638112 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.638113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.638175 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.638619 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.638623 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.638917 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.640545 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.640546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.640547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.640714 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.640911 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.640912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.640912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.640969 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.641255 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.641256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.641316 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.641396 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.641496 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 17:52:11.663630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34365
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:11.747092 8259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:11.747304 8259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:11.747328 8259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:11.748590 8259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:11.748636 8259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:11.748657 8259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 17:52:11.748675 8259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:11.750182 8259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:11.750439 8259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:11.750679 8259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:11.755807 8264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.755771 8267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:11.755792 8265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:11.755867 8259 server_base.cc:1034] running on GCE node
I20260928 17:52:11.756211 8259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:11.756448 8259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:11.757642 8259 hybrid_clock.cc:648] HybridClock initialized: now 1790617931757626 us; error 47 us; skew 500 ppm
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:11.760330 8259 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 17:52:11.760807 8259 webserver.cc:460] Webserver started at http://127.0.98.3:45771/ using document root <none> and password file <none>
I20260928 17:52:11.761034 8259 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:11.761080 8259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:11.761178 8259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:11.762283 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b346bf7cbe44193b3358b1f62fecced"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "cfc4ae2885831fa6db4c368806f926d5"
server_key_iv: "0c055de49e5e1b658a4c13e57fcea0ff"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.762609 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b346bf7cbe44193b3358b1f62fecced"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "cfc4ae2885831fa6db4c368806f926d5"
server_key_iv: "0c055de49e5e1b658a4c13e57fcea0ff"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.764047 8259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:11.764899 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.765107 8259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:11.765201 8259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "4b346bf7cbe44193b3358b1f62fecced"
format_stamp: "Formatted at 2026-09-28 17:52:11 on dist-test-slave-zqlt"
server_key: "cfc4ae2885831fa6db4c368806f926d5"
server_key_iv: "0c055de49e5e1b658a4c13e57fcea0ff"
server_key_version: "encryptionkey@0"
I20260928 17:52:11.765280 8259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:11.782181 8259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:11.784607 8259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:11.784763 8259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:11.785475 8259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:11.785538 8259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:11.785589 8259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:11.785637 8259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:11.793716 8259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34045
I20260928 17:52:11.793817 8389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34045 every 8 connection(s)
I20260928 17:52:11.794158 8259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:11.796298 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.785805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50405 (local address 127.0.98.62:34365)
0928 17:52:11.785859 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.785861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.786124 (+ 263us) server_negotiation.cc:407] Connection header received
0928 17:52:11.786914 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.786916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.786927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.786974 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.788070 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.788698 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.790579 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.790765 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.790849 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.793373 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.793375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.793375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.793384 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.795010 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.795445 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.795445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.795446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.795520 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.795717 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.795717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.795717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.795819 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.795869 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.796164 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.796232 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:52:11.796464 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.785780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.3:50405)
0928 17:52:11.785953 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.786012 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.786797 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.787106 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.787111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.787549 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.787915 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.787924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.788802 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.788803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.790482 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.790486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.790882 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.790883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.791026 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.791536 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.791546 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.793275 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.795086 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.795091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.795092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.795379 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.795579 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.795580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.795580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.795676 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.795867 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.795871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.795983 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.796143 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.796273 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 17:52:11.796909 8390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34365
I20260928 17:52:11.797035 8390 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:11.797221 8390 heartbeater.cc:511] Master 127.0.98.62:34365 requested a full tablet report, sending...
I20260928 17:52:11.797902 7906 ts_manager.cc:194] Registered new tserver with Master: 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045)
I20260928 17:52:11.798491 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50405
I20260928 17:52:11.798751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8259
I20260928 17:52:11.798836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:11.799012 392 external_mini_cluster.cc:1428] Setting key e5ee8402afa9358cf1661ca22cd30cff
I20260928 17:52:11.799849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:52:11.804673 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.797890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56541 (local address 127.0.98.62:34365)
0928 17:52:11.797927 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.797928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.798033 (+ 105us) server_negotiation.cc:407] Connection header received
0928 17:52:11.798107 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.798108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.798116 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.798160 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.798512 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.799264 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.800468 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.800646 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.800704 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.801772 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.801773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.801773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.801778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.803426 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.803829 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.803830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.803830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.803875 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.804188 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.804188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.804189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.804291 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.804342 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.804535 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.804609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:52:11.804927 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.797835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.98.3:56541)
0928 17:52:11.797969 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.797982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.798048 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.798177 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.798178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.798368 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:11.798442 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.798444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.799379 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.799380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.800386 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.800389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.800722 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.800723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.800800 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.801202 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:11.801207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:11.801686 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:11.803506 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.803507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.803507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.803758 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.803949 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:11.803950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:11.803951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:11.804079 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:11.804358 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:11.804359 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:11.804410 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.804507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.804626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:52:11.822769 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:11.817414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.98.62:34365)
0928 17:52:11.817470 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.817472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.817483 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:11.817571 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.817572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.817583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.817644 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.818220 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.818699 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.819463 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.819643 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.819737 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.822647 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59940: BlockingRecv error: recv got EOF from 127.0.0.1:59940 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:52:11 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:11.829939 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.823394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.98.62:34365)
0928 17:52:11.823450 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.823452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.823538 (+ 86us) server_negotiation.cc:407] Connection header received
0928 17:52:11.823609 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.823610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.823619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.823690 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:11.824237 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.824693 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.825365 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.825549 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.825628 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.827657 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:11.827658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:11.827658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:11.827666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:11.828913 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.829268 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.829269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.829269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.829320 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:11.829534 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:11.829534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:11.829535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:11.829633 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:11.829707 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.829820 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.829876 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:52:11.831684 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:11.833359 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:11.840456 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.835653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34001 (local address 127.0.0.1:39040)
0928 17:52:11.835772 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.835857 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.836035 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.836691 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.836694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.836697 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:11.836807 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.836809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.838121 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.838122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.839149 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.839151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.840051 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.840052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.840205 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840224 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.840299 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.840354 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":24,"thread_start_us":75,"threads_started":1}
I20260928 17:52:11.840474 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.835744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.0.98.2:34001)
0928 17:52:11.836318 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.836323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.836335 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:52:11.836561 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.836564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.836584 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.836872 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:11.836934 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.838030 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.839239 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.839983 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.840051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840161 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.840325 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.840367 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":99,"threads_started":1}
I20260928 17:52:11.840466 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.836155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.0.1:50862)
0928 17:52:11.836463 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.836472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.836531 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.836743 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.836743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.836746 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:11.836827 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.836829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.837742 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.837743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.838965 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.838967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.840135 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.840136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.840209 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840227 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.840312 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.840359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260928 17:52:11.840786 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.836248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.0.98.1:46139)
0928 17:52:11.836467 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.836471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.836590 (+ 119us) server_negotiation.cc:407] Connection header received
0928 17:52:11.836635 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.836637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.836653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.836746 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:11.836894 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.837667 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.839110 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.839875 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.839957 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840075 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.840619 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.840666 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260928 17:52:11.841153 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.835895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.0.1:41362)
0928 17:52:11.836061 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.836072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.836162 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.836797 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.836798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.836800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:11.836843 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.836844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.839220 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.839220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.839865 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.839866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.840907 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.840908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.840971 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840975 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.841047 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.841088 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260928 17:52:11.841393 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.836066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.98.3:34045)
0928 17:52:11.836642 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.836648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.836657 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:11.836692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.836694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.836707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.836911 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:11.836967 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.838040 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.839948 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.840733 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.840798 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.840875 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.841255 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.841300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":109,"threads_started":1}
I20260928 17:52:11.841568 8184 tablet_service.cc:1467] Processing CreateTablet for tablet ce186a743c8947149eacfa280296f313 (DEFAULT_TABLE table=test-table [id=91e19abb7beb4f1e85ede28ba0f10211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.841966 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce186a743c8947149eacfa280296f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.841871 8044 tablet_service.cc:1467] Processing CreateTablet for tablet ce186a743c8947149eacfa280296f313 (DEFAULT_TABLE table=test-table [id=91e19abb7beb4f1e85ede28ba0f10211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.841933 8324 tablet_service.cc:1467] Processing CreateTablet for tablet ce186a743c8947149eacfa280296f313 (DEFAULT_TABLE table=test-table [id=91e19abb7beb4f1e85ede28ba0f10211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.842216 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce186a743c8947149eacfa280296f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.842214 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce186a743c8947149eacfa280296f313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.845381 8418 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Bootstrap starting.
I20260928 17:52:11.845938 8419 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Bootstrap starting.
I20260928 17:52:11.846166 8418 tablet_bootstrap.cc:654] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.846412 8420 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Bootstrap starting.
I20260928 17:52:11.846601 8419 tablet_bootstrap.cc:654] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.846617 8418 log.cc:824] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:11.846938 8419 log.cc:824] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:11.847047 8420 tablet_bootstrap.cc:654] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.847376 8420 log.cc:824] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:11.847517 8418 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: No bootstrap required, opened a new log
I20260928 17:52:11.847621 8418 ts_tablet_manager.cc:1397] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:52:11.847704 8419 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: No bootstrap required, opened a new log
I20260928 17:52:11.847769 8419 ts_tablet_manager.cc:1397] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:52:11.848050 8420 tablet_bootstrap.cc:492] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: No bootstrap required, opened a new log
I20260928 17:52:11.848160 8420 ts_tablet_manager.cc:1397] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:52:11.849573 8420 raft_consensus.cc:348] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.849741 8419 raft_consensus.cc:348] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.849741 8418 raft_consensus.cc:348] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.849857 8420 raft_consensus.cc:374] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.849896 8419 raft_consensus.cc:374] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.849896 8418 raft_consensus.cc:374] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.849906 8420 raft_consensus.cc:729] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.849939 8419 raft_consensus.cc:729] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.849939 8418 raft_consensus.cc:729] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.850070 8419 consensus_queue.cc:260] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.850059 8418 consensus_queue.cc:260] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.850092 8420 consensus_queue.cc:260] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.850351 8419 ts_tablet_manager.cc:1428] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:52:11.850351 8418 ts_tablet_manager.cc:1428] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:52:11.850395 8390 heartbeater.cc:503] Master 127.0.98.62:34365 was elected leader, sending a full tablet report...
I20260928 17:52:11.850492 8250 heartbeater.cc:503] Master 127.0.98.62:34365 was elected leader, sending a full tablet report...
I20260928 17:52:11.850651 8420 ts_tablet_manager.cc:1428] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:52:11.850768 8110 heartbeater.cc:503] Master 127.0.98.62:34365 was elected leader, sending a full tablet report...
I20260928 17:52:11.854306 8424 raft_consensus.cc:482] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:11.854408 8424 raft_consensus.cc:504] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.854754 8424 leader_election.cc:288] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001), eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139)
I20260928 17:52:11.857785 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.854975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58229 (local address 127.0.98.2:34001)
0928 17:52:11.855048 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.855050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.855060 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:11.855095 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.855096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.855115 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.855175 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:11.855318 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.855903 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.856954 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.857432 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.857485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.857495 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.857675 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.857703 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 17:52:11.857798 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.854916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34001 (local address 127.0.98.3:58229)
0928 17:52:11.854974 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.854989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.855045 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.855177 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.855178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.855180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:11.855260 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.855263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.855977 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.855978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.856870 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.856873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.857514 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.857515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.857602 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.857606 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.857670 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.857718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:52:11.858045 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.855356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52467 (local address 127.0.98.1:46139)
0928 17:52:11.855413 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.855415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.855422 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:11.855444 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.855444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.855453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.855504 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:11.855671 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.856250 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.857241 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.857687 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.857746 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.857756 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.857939 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.857981 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:52:11.858057 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.855016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.98.3:52467)
0928 17:52:11.855202 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:11.855213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:11.855260 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:11.855528 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:11.855529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:11.855531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:11.855595 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.855596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.856327 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.856328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.857156 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:11.857159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.857759 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.857760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:11.857836 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.857841 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:11.857927 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 17:52:11.857965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260928 17:52:11.858294 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce186a743c8947149eacfa280296f313" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34" is_pre_election: true
I20260928 17:52:11.858343 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce186a743c8947149eacfa280296f313" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed" is_pre_election: true
I20260928 17:52:11.858456 8064 raft_consensus.cc:2454] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:11.858462 8204 raft_consensus.cc:2454] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:11.858660 8276 leader_election.cc:302] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, fa41f7c541354afe9ba9f923e87c2c34; no voters:
I20260928 17:52:11.858774 8424 raft_consensus.cc:2781] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:11.858829 8424 raft_consensus.cc:482] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:11.858855 8424 raft_consensus.cc:3037] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:11.859972 8424 raft_consensus.cc:504] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.860146 8424 leader_election.cc:288] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Requested vote from peers fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001), eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139)
I20260928 17:52:11.860286 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce186a743c8947149eacfa280296f313" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34"
I20260928 17:52:11.860301 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce186a743c8947149eacfa280296f313" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed"
I20260928 17:52:11.860384 8204 raft_consensus.cc:3037] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:11.860383 8064 raft_consensus.cc:3037] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:11.861106 8064 raft_consensus.cc:2454] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:11.861109 8204 raft_consensus.cc:2454] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:11.861264 8276 leader_election.cc:302] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, eb66ff1dd84247758e80314c6cd204ed; no voters:
I20260928 17:52:11.861361 8424 raft_consensus.cc:2781] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:11.861513 8424 raft_consensus.cc:686] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 1 LEADER]: Becoming Leader. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Running, Role: LEADER
I20260928 17:52:11.861619 8424 consensus_queue.cc:237] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } }
I20260928 17:52:11.862543 7906 catalog_manager.cc:5495] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20260928 17:52:11.869904 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:11.867494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.0.98.3:34045)
0928 17:52:11.867569 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:11.867571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:11.867580 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:11.867655 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:11.867656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:11.867672 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:11.867732 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:11.867863 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.868403 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.869217 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:11.869398 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:11.869481 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:11.869615 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:11.869709 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:11.869770 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:11.869800 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 17:52:11.869818 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 17:52:11.871852 8204 raft_consensus.cc:1261] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:11.871838 8064 raft_consensus.cc:1261] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:11.872279 8424 consensus_queue.cc:1035] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:11.872552 8428 consensus_queue.cc:1035] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:11.875231 8432 mvcc.cc:205] Tried to move back new op lower bound from 7334371048944418816 to 7334371048906084352. Current Snapshot: MvccSnapshot[applied={T|T < 7334371048944418816}]
I20260928 17:52:11.877941 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:11.878051 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:11.878819 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Sending DeleteTablet for 3 replicas of tablet ce186a743c8947149eacfa280296f313
I20260928 17:52:11.879163 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet ce186a743c8947149eacfa280296f313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39040
I20260928 17:52:11.879220 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet ce186a743c8947149eacfa280296f313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260928 17:52:11.879293 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet ce186a743c8947149eacfa280296f313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260928 17:52:11.879493 8443 tablet_replica.cc:330] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: stopping tablet replica
I20260928 17:52:11.879575 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 17:52:11.879637 8443 raft_consensus.cc:2229] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 1 LEADER]: Raft consensus shutting down.
W20260928 17:52:11.879725 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:11.879899 8442 tablet_replica.cc:330] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: stopping tablet replica
I20260928 17:52:11.879940 8443 raft_consensus.cc:2258] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:11.880056 8442 raft_consensus.cc:2229] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:11.880235 8442 raft_consensus.cc:2258] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:11.880759 8442 ts_tablet_manager.cc:1905] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:11.880781 8443 ts_tablet_manager.cc:1905] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:11.881122 8044 tablet_service.cc:1467] Processing CreateTablet for tablet c2d177b478bb43d2bc429ea745e02e4f (DEFAULT_TABLE table=test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.881134 8184 tablet_service.cc:1467] Processing CreateTablet for tablet c2d177b478bb43d2bc429ea745e02e4f (DEFAULT_TABLE table=test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.881239 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d177b478bb43d2bc429ea745e02e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.881264 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d177b478bb43d2bc429ea745e02e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.882735 8420 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Bootstrap starting.
I20260928 17:52:11.882853 8418 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Bootstrap starting.
I20260928 17:52:11.882947 8442 ts_tablet_manager.cc:1918] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:11.883078 8442 log.cc:1196] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/ce186a743c8947149eacfa280296f313
I20260928 17:52:11.883154 8420 tablet_bootstrap.cc:654] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.883538 8442 ts_tablet_manager.cc:1939] T ce186a743c8947149eacfa280296f313 P eb66ff1dd84247758e80314c6cd204ed: Deleting consensus metadata
I20260928 17:52:11.883610 8418 tablet_bootstrap.cc:654] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.883697 8324 tablet_service.cc:1467] Processing CreateTablet for tablet c2d177b478bb43d2bc429ea745e02e4f (DEFAULT_TABLE table=test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:11.883808 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d177b478bb43d2bc429ea745e02e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:11.884225 7892 catalog_manager.cc:4841] TS eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139): tablet ce186a743c8947149eacfa280296f313 (table test-table [id=91e19abb7beb4f1e85ede28ba0f10211]) successfully deleted
I20260928 17:52:11.885068 8419 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Bootstrap starting.
I20260928 17:52:11.885473 8419 tablet_bootstrap.cc:654] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:11.885710 8420 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: No bootstrap required, opened a new log
I20260928 17:52:11.885751 8420 ts_tablet_manager.cc:1397] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 17:52:11.885901 8420 raft_consensus.cc:348] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.885979 8420 raft_consensus.cc:374] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.886011 8420 raft_consensus.cc:729] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.886057 8420 consensus_queue.cc:260] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.886083 8419 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: No bootstrap required, opened a new log
I20260928 17:52:11.886133 8419 ts_tablet_manager.cc:1397] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:11.886147 8420 ts_tablet_manager.cc:1428] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:11.886297 8419 raft_consensus.cc:348] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.886387 8419 raft_consensus.cc:374] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.886420 8419 raft_consensus.cc:729] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.886482 8419 consensus_queue.cc:260] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.886591 8419 ts_tablet_manager.cc:1428] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:11.887144 8443 ts_tablet_manager.cc:1918] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:11.887225 8443 log.cc:1196] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/ce186a743c8947149eacfa280296f313
I20260928 17:52:11.887513 8443 ts_tablet_manager.cc:1939] T ce186a743c8947149eacfa280296f313 P 4b346bf7cbe44193b3358b1f62fecced: Deleting consensus metadata
I20260928 17:52:11.887945 7892 catalog_manager.cc:4841] TS 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045): tablet ce186a743c8947149eacfa280296f313 (table test-table [id=91e19abb7beb4f1e85ede28ba0f10211]) successfully deleted
I20260928 17:52:11.888214 8441 tablet_replica.cc:330] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: stopping tablet replica
I20260928 17:52:11.888338 8441 raft_consensus.cc:2229] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:11.888473 8441 raft_consensus.cc:2258] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:11.888495 8418 tablet_bootstrap.cc:492] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: No bootstrap required, opened a new log
I20260928 17:52:11.888558 8418 ts_tablet_manager.cc:1397] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 17:52:11.888722 8418 raft_consensus.cc:348] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.888808 8418 raft_consensus.cc:374] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:11.888834 8418 raft_consensus.cc:729] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Initialized, Role: FOLLOWER
I20260928 17:52:11.888876 8418 consensus_queue.cc:260] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:11.888913 8441 ts_tablet_manager.cc:1905] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:11.888974 8418 ts_tablet_manager.cc:1428] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:11.890669 8441 ts_tablet_manager.cc:1918] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:11.890748 8441 log.cc:1196] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/ce186a743c8947149eacfa280296f313
I20260928 17:52:11.890988 8441 ts_tablet_manager.cc:1939] T ce186a743c8947149eacfa280296f313 P fa41f7c541354afe9ba9f923e87c2c34: Deleting consensus metadata
I20260928 17:52:11.891397 7892 catalog_manager.cc:4841] TS fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001): tablet ce186a743c8947149eacfa280296f313 (table test-table [id=91e19abb7beb4f1e85ede28ba0f10211]) successfully deleted
W20260928 17:52:11.974114 8111 tablet.cc:2369] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:52:12.044744 8391 tablet.cc:2369] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:12.123946 8425 raft_consensus.cc:482] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:12.124198 8425 raft_consensus.cc:504] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.124552 8425 leader_election.cc:288] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045)
I20260928 17:52:12.127486 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.124777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.98.2:41093)
0928 17:52:12.124831 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.124843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.124891 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.125080 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.125081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.125083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:12.125181 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.125183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.125899 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.125900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.126683 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.126685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.127223 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.127224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.127304 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.127309 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.127362 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.127394 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:52:12.127513 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.124855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41093 (local address 127.0.98.1:46139)
0928 17:52:12.124918 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.124921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.124934 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:52:12.124966 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.124967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.124977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.125049 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:12.125268 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.125825 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.126778 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.127152 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.127206 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.127214 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.127389 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.127435 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:52:12.127578 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.125003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43963 (local address 127.0.98.3:34045)
0928 17:52:12.125077 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.125078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.125111 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:52:12.125186 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.125187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.125196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.125251 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:12.125424 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.125894 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.126845 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.127218 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.127287 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.127297 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.127465 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.127506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:52:12.127810 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.124896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.98.2:43963)
0928 17:52:12.125051 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.125063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.125119 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.125266 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.125267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.125268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:12.125352 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.125354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.125975 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.125976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.126749 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.126752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.127305 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.127305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.127384 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.127388 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.127442 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.127479 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260928 17:52:12.127908 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d177b478bb43d2bc429ea745e02e4f" candidate_uuid: "fa41f7c541354afe9ba9f923e87c2c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed" is_pre_election: true
I20260928 17:52:12.128012 8064 raft_consensus.cc:2454] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa41f7c541354afe9ba9f923e87c2c34 in term 0.
I20260928 17:52:12.128046 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d177b478bb43d2bc429ea745e02e4f" candidate_uuid: "fa41f7c541354afe9ba9f923e87c2c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b346bf7cbe44193b3358b1f62fecced" is_pre_election: true
I20260928 17:52:12.128190 8344 raft_consensus.cc:2454] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa41f7c541354afe9ba9f923e87c2c34 in term 0.
I20260928 17:52:12.128182 8136 leader_election.cc:302] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb66ff1dd84247758e80314c6cd204ed, fa41f7c541354afe9ba9f923e87c2c34; no voters:
I20260928 17:52:12.128325 8425 raft_consensus.cc:2781] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:12.128376 8425 raft_consensus.cc:482] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:12.128391 8425 raft_consensus.cc:3037] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.129037 8425 raft_consensus.cc:504] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.129137 8425 leader_election.cc:288] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [CANDIDATE]: Term 1 election: Requested vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045)
I20260928 17:52:12.129316 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d177b478bb43d2bc429ea745e02e4f" candidate_uuid: "fa41f7c541354afe9ba9f923e87c2c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed"
I20260928 17:52:12.129333 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d177b478bb43d2bc429ea745e02e4f" candidate_uuid: "fa41f7c541354afe9ba9f923e87c2c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b346bf7cbe44193b3358b1f62fecced"
I20260928 17:52:12.129402 8064 raft_consensus.cc:3037] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.129401 8344 raft_consensus.cc:3037] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.130029 8344 raft_consensus.cc:2454] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa41f7c541354afe9ba9f923e87c2c34 in term 1.
I20260928 17:52:12.130034 8064 raft_consensus.cc:2454] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa41f7c541354afe9ba9f923e87c2c34 in term 1.
I20260928 17:52:12.130205 8136 leader_election.cc:302] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb66ff1dd84247758e80314c6cd204ed, fa41f7c541354afe9ba9f923e87c2c34; no voters:
I20260928 17:52:12.130326 8425 raft_consensus.cc:2781] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:12.130445 8425 raft_consensus.cc:686] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 1 LEADER]: Becoming Leader. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Running, Role: LEADER
I20260928 17:52:12.130522 8425 consensus_queue.cc:237] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.131175 7905 catalog_manager.cc:5495] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fa41f7c541354afe9ba9f923e87c2c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } health_report { overall_health: UNKNOWN } } }
W20260928 17:52:12.132275 8251 tablet.cc:2369] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:12.195545 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.188856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34365 (local address 127.0.0.1:59958)
0928 17:52:12.189043 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.189053 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.189134 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.189265 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.189266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.189574 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:12.189623 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.189625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.190279 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.190280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.190800 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.190801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.191154 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.191154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.191194 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.191685 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:12.191689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:12.193085 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:12.194491 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:12.194499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:12.194501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:12.194706 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:12.194959 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:12.194960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:12.194960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:12.195015 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:12.195276 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:12.195282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:12.195323 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.195376 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.195469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260928 17:52:12.195608 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.188900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.0.98.62:34365)
0928 17:52:12.188966 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.188968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.189118 (+ 150us) server_negotiation.cc:407] Connection header received
0928 17:52:12.189186 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.189187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.189197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.189249 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:12.189714 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.190205 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.190884 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.191088 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.191147 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.193182 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:12.193183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:12.193184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:12.193190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:12.194416 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:12.194787 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:12.194788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:12.194789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:12.194845 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:12.195089 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:12.195090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:12.195090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:12.195213 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:12.195279 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.195407 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.195521 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":37}
I20260928 17:52:12.197793 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 17:52:12.197992 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:12.199229 8044 tablet_service.cc:1467] Processing CreateTablet for tablet a66f603eb09b4125ab265896b912d0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41993a2cee9049e08879e5c57279bc86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:52:12.199388 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66f603eb09b4125ab265896b912d0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.200246 8324 tablet_service.cc:1467] Processing CreateTablet for tablet a66f603eb09b4125ab265896b912d0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41993a2cee9049e08879e5c57279bc86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:52:12.200392 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66f603eb09b4125ab265896b912d0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.200577 8420 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: Bootstrap starting.
I20260928 17:52:12.200942 8420 tablet_bootstrap.cc:654] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.201504 8420 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: No bootstrap required, opened a new log
I20260928 17:52:12.201542 8420 ts_tablet_manager.cc:1397] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:12.201653 8420 raft_consensus.cc:348] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.201704 8420 raft_consensus.cc:374] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.201718 8420 raft_consensus.cc:729] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.201773 8420 consensus_queue.cc:260] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.201843 8419 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: Bootstrap starting.
I20260928 17:52:12.201897 8420 ts_tablet_manager.cc:1428] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.201946 8184 tablet_service.cc:1467] Processing CreateTablet for tablet a66f603eb09b4125ab265896b912d0b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41993a2cee9049e08879e5c57279bc86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 17:52:12.202078 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66f603eb09b4125ab265896b912d0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.202168 8419 tablet_bootstrap.cc:654] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.202145 8426 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.202235 8426 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.202778 8419 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: No bootstrap required, opened a new log
I20260928 17:52:12.202836 8419 ts_tablet_manager.cc:1397] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:12.202939 8419 raft_consensus.cc:348] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.203012 8419 raft_consensus.cc:374] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.203032 8419 raft_consensus.cc:729] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.203071 8419 consensus_queue.cc:260] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.203167 8419 ts_tablet_manager.cc:1428] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:12.203263 8436 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.203336 8436 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.203599 8418 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: Bootstrap starting.
I20260928 17:52:12.203985 8418 tablet_bootstrap.cc:654] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.204581 8418 tablet_bootstrap.cc:492] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: No bootstrap required, opened a new log
I20260928 17:52:12.204622 8418 ts_tablet_manager.cc:1397] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:12.204725 8418 raft_consensus.cc:348] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.204799 8418 raft_consensus.cc:374] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.204813 8418 raft_consensus.cc:729] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.204864 8418 consensus_queue.cc:260] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.204959 8418 ts_tablet_manager.cc:1428] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.205039 8425 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.205142 8425 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.333341 8436 raft_consensus.cc:482] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:12.333452 8436 raft_consensus.cc:504] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.333617 8436 leader_election.cc:288] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001)
I20260928 17:52:12.333825 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66f603eb09b4125ab265896b912d0b9" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed" is_pre_election: true
I20260928 17:52:12.333890 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66f603eb09b4125ab265896b912d0b9" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34" is_pre_election: true
I20260928 17:52:12.333949 8064 raft_consensus.cc:2454] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:12.333968 8204 raft_consensus.cc:2454] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:12.334125 8276 leader_election.cc:302] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, fa41f7c541354afe9ba9f923e87c2c34; no voters:
I20260928 17:52:12.334311 8436 raft_consensus.cc:2781] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:12.334358 8436 raft_consensus.cc:482] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:12.334373 8436 raft_consensus.cc:3037] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.335215 8436 raft_consensus.cc:504] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.335341 8436 leader_election.cc:288] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Requested vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001)
I20260928 17:52:12.335549 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66f603eb09b4125ab265896b912d0b9" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34"
I20260928 17:52:12.335531 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66f603eb09b4125ab265896b912d0b9" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed"
I20260928 17:52:12.335626 8064 raft_consensus.cc:3037] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.335626 8204 raft_consensus.cc:3037] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.336444 8204 raft_consensus.cc:2454] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:12.336444 8064 raft_consensus.cc:2454] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:12.336650 8276 leader_election.cc:302] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, eb66ff1dd84247758e80314c6cd204ed; no voters:
I20260928 17:52:12.336746 8436 raft_consensus.cc:2781] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:12.336798 8436 raft_consensus.cc:686] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [term 1 LEADER]: Becoming Leader. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Running, Role: LEADER
I20260928 17:52:12.336854 8436 consensus_queue.cc:237] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.337173 8424 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: TxnStatusTablet state changed. Reason: New leader 4b346bf7cbe44193b3358b1f62fecced. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.337273 8424 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:52:12.337420 8460 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 17:52:12.337567 8460 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 17:52:12.337554 7905 catalog_manager.cc:5495] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } health_report { overall_health: HEALTHY } } }
I20260928 17:52:12.351660 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.349348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.0.98.3:34045)
0928 17:52:12.349399 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.349401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.349411 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:12.349501 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.349502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.349517 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.349571 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.349693 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.350175 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.350947 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.351149 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.351220 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.351397 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.351436 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.351495 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.351558 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.351581 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:52:12.351678 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.349280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.0.1:41368)
0928 17:52:12.349355 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.349369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.349425 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.349578 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.349579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.349580 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:52:12.349632 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.349635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.350271 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.350272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.350860 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.350861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.351231 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.351232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.351304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.351318 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.351501 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.351510 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.351557 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.351609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:52:12.352878 8204 raft_consensus.cc:1261] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.352928 8064 raft_consensus.cc:1261] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.353122 8436 consensus_queue.cc:1035] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.353215 8424 consensus_queue.cc:1035] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.353713 8426 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: TxnStatusTablet state changed. Reason: New leader 4b346bf7cbe44193b3358b1f62fecced. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.353734 8425 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: TxnStatusTablet state changed. Reason: New leader 4b346bf7cbe44193b3358b1f62fecced. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.353827 8426 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.353834 8425 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.354636 8436 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.354729 8436 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:52:12.354775 8436 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.354820 8436 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P 4b346bf7cbe44193b3358b1f62fecced: This TxnStatusTablet replica's current role is: LEADER
I20260928 17:52:12.354836 8433 mvcc.cc:205] Tried to move back new op lower bound from 7334371050915594240 to 7334371050852319232. Current Snapshot: MvccSnapshot[applied={T|T < 7334371050915594240}]
I20260928 17:52:12.354868 8431 mvcc.cc:205] Tried to move back new op lower bound from 7334371050915594240 to 7334371050852319232. Current Snapshot: MvccSnapshot[applied={T|T < 7334371050915594240}]
I20260928 17:52:12.354971 8425 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.355043 8425 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P fa41f7c541354afe9ba9f923e87c2c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.355103 8426 tablet_replica.cc:439] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } } }
I20260928 17:52:12.355168 8426 tablet_replica.cc:442] T a66f603eb09b4125ab265896b912d0b9 P eb66ff1dd84247758e80314c6cd204ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 17:52:12.359284 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.356809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.0.98.2:34001)
0928 17:52:12.356853 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.356855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.356864 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:12.356934 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.356935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.356955 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.357045 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.357158 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.357633 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.358466 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.358677 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.358751 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.358914 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.359044 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.359137 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.359177 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.359201 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:12.364557 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.361854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59837 (local address 127.0.98.3:34045)
0928 17:52:12.361912 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.361914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.362048 (+ 134us) server_negotiation.cc:407] Connection header received
0928 17:52:12.362108 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.362109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.362122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.362183 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.362277 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.362982 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.363718 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.364019 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.364118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.364315 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.364351 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.364408 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.364453 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.364477 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:52:12.364758 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.361770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.98.2:59837)
0928 17:52:12.361945 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.361956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.362047 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.362184 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.362185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.362186 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:52:12.362238 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.362239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.363066 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.363067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.363644 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.363645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.364162 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.364163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.364218 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.364232 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.364413 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.364415 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.364449 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.364485 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 17:52:12.367290 8344 raft_consensus.cc:1261] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Refusing update from remote peer fa41f7c541354afe9ba9f923e87c2c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.367548 8064 raft_consensus.cc:1261] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Refusing update from remote peer fa41f7c541354afe9ba9f923e87c2c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.367587 8448 consensus_queue.cc:1035] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.367777 8448 consensus_queue.cc:1035] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.376786 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.374012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34001 (local address 127.0.98.3:42509)
0928 17:52:12.374130 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.374141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.374198 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.374319 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.374320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.374321 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:52:12.374384 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.374386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.375015 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.375016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.375897 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.375899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.376383 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.376384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.376456 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.376472 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.376653 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.376654 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.376683 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.376722 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260928 17:52:12.376786 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.374048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42509 (local address 127.0.98.2:34001)
0928 17:52:12.374097 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.374099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.374201 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:52:12.374251 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.374252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.374262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.374316 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.374452 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.374934 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.375973 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.376310 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.376382 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.376561 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.376603 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.376654 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.376691 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.376710 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:52:12.390226 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.390342 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.390926 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Sending DeleteTablet for 3 replicas of tablet c2d177b478bb43d2bc429ea745e02e4f
I20260928 17:52:12.391201 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d177b478bb43d2bc429ea745e02e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260928 17:52:12.391242 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d177b478bb43d2bc429ea745e02e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260928 17:52:12.391561 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d177b478bb43d2bc429ea745e02e4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39040
I20260928 17:52:12.391728 8476 tablet_replica.cc:330] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: stopping tablet replica
I20260928 17:52:12.391784 8476 raft_consensus.cc:2229] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.391819 8476 raft_consensus.cc:2258] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.392053 8476 ts_tablet_manager.cc:1905] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.393738 8477 tablet_replica.cc:330] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: stopping tablet replica
I20260928 17:52:12.393822 8477 raft_consensus.cc:2229] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:12.393900 8477 raft_consensus.cc:2258] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.394125 8477 ts_tablet_manager.cc:1905] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.395679 8477 ts_tablet_manager.cc:1918] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:52:12.395742 8477 log.cc:1196] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/c2d177b478bb43d2bc429ea745e02e4f
I20260928 17:52:12.395961 8477 ts_tablet_manager.cc:1939] T c2d177b478bb43d2bc429ea745e02e4f P fa41f7c541354afe9ba9f923e87c2c34: Deleting consensus metadata
I20260928 17:52:12.396281 7892 catalog_manager.cc:4841] TS fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001): tablet c2d177b478bb43d2bc429ea745e02e4f (table test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]) successfully deleted
I20260928 17:52:12.396430 8476 ts_tablet_manager.cc:1918] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:52:12.396489 8476 log.cc:1196] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/c2d177b478bb43d2bc429ea745e02e4f
I20260928 17:52:12.396713 8476 ts_tablet_manager.cc:1939] T c2d177b478bb43d2bc429ea745e02e4f P eb66ff1dd84247758e80314c6cd204ed: Deleting consensus metadata
I20260928 17:52:12.396988 7892 catalog_manager.cc:4841] TS eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139): tablet c2d177b478bb43d2bc429ea745e02e4f (table test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]) successfully deleted
I20260928 17:52:12.397106 8475 tablet_replica.cc:330] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: stopping tablet replica
I20260928 17:52:12.397166 8475 raft_consensus.cc:2229] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.397207 8475 raft_consensus.cc:2258] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.397504 8475 ts_tablet_manager.cc:1905] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.399052 8475 ts_tablet_manager.cc:1918] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:52:12.399112 8475 log.cc:1196] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/c2d177b478bb43d2bc429ea745e02e4f
I20260928 17:52:12.399314 8475 ts_tablet_manager.cc:1939] T c2d177b478bb43d2bc429ea745e02e4f P 4b346bf7cbe44193b3358b1f62fecced: Deleting consensus metadata
I20260928 17:52:12.399648 7892 catalog_manager.cc:4841] TS 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045): tablet c2d177b478bb43d2bc429ea745e02e4f (table test-table [id=6b3bf9a5e02b4473ba48b6b9e0a29ba2]) successfully deleted
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:52:12.428908 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:12.421748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.98.62:34365)
0928 17:52:12.421794 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.421796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.421959 (+ 163us) server_negotiation.cc:407] Connection header received
0928 17:52:12.422021 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.422022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.422037 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.422110 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:12.422861 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.423636 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.424804 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.425092 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.425178 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.428836 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59968: BlockingRecv error: recv got EOF from 127.0.0.1:59968 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 17:52:12.439016 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.430072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.98.62:34365)
0928 17:52:12.430109 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.430111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.430211 (+ 100us) server_negotiation.cc:407] Connection header received
0928 17:52:12.430305 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.430305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.430314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.430344 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:12.430978 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.431748 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.432841 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.433120 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.433161 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.436068 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:12.436070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:12.436070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:12.436079 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:12.437754 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:12.438145 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:12.438146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:12.438146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:12.438202 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:12.438387 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:12.438388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:12.438388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:12.438530 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:12.438603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.438804 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.438886 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:52:12.443511 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:12.443720 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:12.445123 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 26c61bee20d94b17a31119b4981402ea (DEFAULT_TABLE table=test-table [id=c17eaa3de849439d9b1bd9df81992ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.445120 8184 tablet_service.cc:1467] Processing CreateTablet for tablet 26c61bee20d94b17a31119b4981402ea (DEFAULT_TABLE table=test-table [id=c17eaa3de849439d9b1bd9df81992ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.445261 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c61bee20d94b17a31119b4981402ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.445261 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c61bee20d94b17a31119b4981402ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.446588 8419 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Bootstrap starting.
I20260928 17:52:12.446707 8418 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Bootstrap starting.
I20260928 17:52:12.446961 8419 tablet_bootstrap.cc:654] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.447139 8418 tablet_bootstrap.cc:654] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.447570 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 26c61bee20d94b17a31119b4981402ea (DEFAULT_TABLE table=test-table [id=c17eaa3de849439d9b1bd9df81992ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.447731 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c61bee20d94b17a31119b4981402ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.447969 8419 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: No bootstrap required, opened a new log
I20260928 17:52:12.448015 8419 ts_tablet_manager.cc:1397] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:12.448190 8419 raft_consensus.cc:348] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.448257 8419 raft_consensus.cc:374] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.448277 8419 raft_consensus.cc:729] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.448318 8419 consensus_queue.cc:260] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.448401 8419 ts_tablet_manager.cc:1428] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.449290 8420 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Bootstrap starting.
I20260928 17:52:12.449731 8420 tablet_bootstrap.cc:654] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.450052 8418 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: No bootstrap required, opened a new log
I20260928 17:52:12.450107 8418 ts_tablet_manager.cc:1397] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:52:12.450263 8418 raft_consensus.cc:348] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.450345 8418 raft_consensus.cc:374] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.450364 8418 raft_consensus.cc:729] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.450407 8418 consensus_queue.cc:260] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.450495 8418 ts_tablet_manager.cc:1428] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.452596 8420 tablet_bootstrap.cc:492] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: No bootstrap required, opened a new log
I20260928 17:52:12.452657 8420 ts_tablet_manager.cc:1397] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 17:52:12.452811 8420 raft_consensus.cc:348] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.452878 8420 raft_consensus.cc:374] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.452898 8420 raft_consensus.cc:729] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.452950 8420 consensus_queue.cc:260] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.453043 8420 ts_tablet_manager.cc:1428] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.617553 8426 raft_consensus.cc:482] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:12.617669 8426 raft_consensus.cc:504] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.618088 8426 leader_election.cc:288] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001), 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045)
I20260928 17:52:12.621361 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.618373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46271 (local address 127.0.98.2:34001)
0928 17:52:12.618465 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.618467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.618564 (+ 97us) server_negotiation.cc:407] Connection header received
0928 17:52:12.618721 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.618723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.618735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.618794 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:12.618964 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.619509 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.620608 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.621010 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.621067 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.621077 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.621231 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.621275 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 17:52:12.621466 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.618606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52619 (local address 127.0.98.3:34045)
0928 17:52:12.618652 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.618654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.618713 (+ 59us) server_negotiation.cc:407] Connection header received
0928 17:52:12.618803 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.618804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.618813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.618874 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:12.619046 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.619503 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.620673 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.621085 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.621163 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.621173 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.621355 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.621401 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:52:12.621351 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.618300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34001 (local address 127.0.98.1:46271)
0928 17:52:12.618453 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.618465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.618567 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.618805 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.618806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.618808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:12.618883 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.618885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.619626 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.619627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.620515 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.620518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.621087 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.621088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.621154 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.621159 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.621213 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.621255 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":47,"thread_start_us":79,"threads_started":1}
I20260928 17:52:12.621477 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.618512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.98.1:52619)
0928 17:52:12.618629 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.618638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.618724 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.618885 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.618886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.618887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:12.618965 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.618967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.619620 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.619621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.620549 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.620551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.621174 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.621175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.621270 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.621274 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.621334 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.621370 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260928 17:52:12.621834 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c61bee20d94b17a31119b4981402ea" candidate_uuid: "eb66ff1dd84247758e80314c6cd204ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b346bf7cbe44193b3358b1f62fecced" is_pre_election: true
I20260928 17:52:12.621820 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c61bee20d94b17a31119b4981402ea" candidate_uuid: "eb66ff1dd84247758e80314c6cd204ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34" is_pre_election: true
I20260928 17:52:12.621938 8344 raft_consensus.cc:2454] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb66ff1dd84247758e80314c6cd204ed in term 0.
I20260928 17:52:12.621941 8204 raft_consensus.cc:2454] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb66ff1dd84247758e80314c6cd204ed in term 0.
I20260928 17:52:12.622151 7996 leader_election.cc:302] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb66ff1dd84247758e80314c6cd204ed, fa41f7c541354afe9ba9f923e87c2c34; no voters:
I20260928 17:52:12.622284 8426 raft_consensus.cc:2781] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:12.622324 8426 raft_consensus.cc:482] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:12.622354 8426 raft_consensus.cc:3037] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.623040 8426 raft_consensus.cc:504] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.623172 8426 leader_election.cc:288] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [CANDIDATE]: Term 1 election: Requested vote from peers fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001), 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045)
I20260928 17:52:12.623335 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c61bee20d94b17a31119b4981402ea" candidate_uuid: "eb66ff1dd84247758e80314c6cd204ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34"
I20260928 17:52:12.623415 8204 raft_consensus.cc:3037] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.623356 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c61bee20d94b17a31119b4981402ea" candidate_uuid: "eb66ff1dd84247758e80314c6cd204ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b346bf7cbe44193b3358b1f62fecced"
I20260928 17:52:12.623450 8344 raft_consensus.cc:3037] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.624046 8344 raft_consensus.cc:2454] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb66ff1dd84247758e80314c6cd204ed in term 1.
I20260928 17:52:12.624135 8204 raft_consensus.cc:2454] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb66ff1dd84247758e80314c6cd204ed in term 1.
I20260928 17:52:12.624253 7996 leader_election.cc:302] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, eb66ff1dd84247758e80314c6cd204ed; no voters:
I20260928 17:52:12.624380 8426 raft_consensus.cc:2781] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:12.624533 8426 raft_consensus.cc:686] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 1 LEADER]: Becoming Leader. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Running, Role: LEADER
I20260928 17:52:12.624625 8426 consensus_queue.cc:237] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.625341 7905 catalog_manager.cc:5495] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed reported cstate change: term changed from 0 to 1, leader changed from <none> to eb66ff1dd84247758e80314c6cd204ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "eb66ff1dd84247758e80314c6cd204ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } health_report { overall_health: UNKNOWN } } }
I20260928 17:52:12.634264 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.631355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.98.1:46139)
0928 17:52:12.631541 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.631543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.631552 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:12.631589 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.631590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.631613 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.631692 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.631844 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.632638 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.633388 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.633701 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.633779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.633889 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.634018 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.634098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.634156 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.634180 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":54,"threads_started":1}
I20260928 17:52:12.635282 8204 raft_consensus.cc:1261] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Refusing update from remote peer eb66ff1dd84247758e80314c6cd204ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.635344 8344 raft_consensus.cc:1261] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Refusing update from remote peer eb66ff1dd84247758e80314c6cd204ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.635588 8426 consensus_queue.cc:1035] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.635746 8495 consensus_queue.cc:1035] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.639434 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.639521 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.640116 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Sending DeleteTablet for 3 replicas of tablet 26c61bee20d94b17a31119b4981402ea
I20260928 17:52:12.640329 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c61bee20d94b17a31119b4981402ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260928 17:52:12.640429 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c61bee20d94b17a31119b4981402ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39040
I20260928 17:52:12.640507 8476 tablet_replica.cc:330] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: stopping tablet replica
I20260928 17:52:12.640553 8476 raft_consensus.cc:2229] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:12.640535 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 17:52:12.640630 8476 raft_consensus.cc:2258] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 17:52:12.640654 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:12.640703 8477 tablet_replica.cc:330] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: stopping tablet replica
I20260928 17:52:12.640753 8477 raft_consensus.cc:2229] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.640796 8477 raft_consensus.cc:2258] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.641047 8476 ts_tablet_manager.cc:1905] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.641361 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 26c61bee20d94b17a31119b4981402ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260928 17:52:12.641451 8475 tablet_replica.cc:330] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: stopping tablet replica
I20260928 17:52:12.641492 8475 raft_consensus.cc:2229] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.641530 8475 raft_consensus.cc:2258] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.641688 8477 ts_tablet_manager.cc:1905] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.641963 8475 ts_tablet_manager.cc:1905] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.642650 8476 ts_tablet_manager.cc:1918] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:12.642709 8476 log.cc:1196] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/26c61bee20d94b17a31119b4981402ea
I20260928 17:52:12.642922 8476 ts_tablet_manager.cc:1939] T 26c61bee20d94b17a31119b4981402ea P eb66ff1dd84247758e80314c6cd204ed: Deleting consensus metadata
I20260928 17:52:12.643257 8477 ts_tablet_manager.cc:1918] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:12.643306 8477 log.cc:1196] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/26c61bee20d94b17a31119b4981402ea
I20260928 17:52:12.643329 7892 catalog_manager.cc:4841] TS eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139): tablet 26c61bee20d94b17a31119b4981402ea (table test-table [id=c17eaa3de849439d9b1bd9df81992ae1]) successfully deleted
I20260928 17:52:12.643478 8475 ts_tablet_manager.cc:1918] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:12.643528 8475 log.cc:1196] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/26c61bee20d94b17a31119b4981402ea
I20260928 17:52:12.643544 8477 ts_tablet_manager.cc:1939] T 26c61bee20d94b17a31119b4981402ea P fa41f7c541354afe9ba9f923e87c2c34: Deleting consensus metadata
I20260928 17:52:12.643747 8475 ts_tablet_manager.cc:1939] T 26c61bee20d94b17a31119b4981402ea P 4b346bf7cbe44193b3358b1f62fecced: Deleting consensus metadata
I20260928 17:52:12.643893 7892 catalog_manager.cc:4841] TS fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001): tablet 26c61bee20d94b17a31119b4981402ea (table test-table [id=c17eaa3de849439d9b1bd9df81992ae1]) successfully deleted
I20260928 17:52:12.644361 7892 catalog_manager.cc:4841] TS 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045): tablet 26c61bee20d94b17a31119b4981402ea (table test-table [id=c17eaa3de849439d9b1bd9df81992ae1]) successfully deleted
I20260928 17:52:12.644418 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d (DEFAULT_TABLE table=test-table [id=efe3d7cf5ab046fe93157d206672cbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.644418 8184 tablet_service.cc:1467] Processing CreateTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d (DEFAULT_TABLE table=test-table [id=efe3d7cf5ab046fe93157d206672cbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.644515 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.644515 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.645649 8418 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Bootstrap starting.
I20260928 17:52:12.645653 8420 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: Bootstrap starting.
I20260928 17:52:12.645970 8418 tablet_bootstrap.cc:654] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.646000 8420 tablet_bootstrap.cc:654] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.646474 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d (DEFAULT_TABLE table=test-table [id=efe3d7cf5ab046fe93157d206672cbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:12.646529 8418 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: No bootstrap required, opened a new log
I20260928 17:52:12.646569 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:12.646575 8418 ts_tablet_manager.cc:1397] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:12.646682 8418 raft_consensus.cc:348] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.646767 8418 raft_consensus.cc:374] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.646790 8418 raft_consensus.cc:729] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa41f7c541354afe9ba9f923e87c2c34, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.646829 8418 consensus_queue.cc:260] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.646914 8418 ts_tablet_manager.cc:1428] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.647502 8420 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: No bootstrap required, opened a new log
I20260928 17:52:12.647547 8420 ts_tablet_manager.cc:1397] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:52:12.647711 8420 raft_consensus.cc:348] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.647794 8420 raft_consensus.cc:374] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.647814 8420 raft_consensus.cc:729] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb66ff1dd84247758e80314c6cd204ed, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.647855 8420 consensus_queue.cc:260] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.647954 8420 ts_tablet_manager.cc:1428] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.648032 8419 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Bootstrap starting.
I20260928 17:52:12.648406 8419 tablet_bootstrap.cc:654] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:12.648921 8419 tablet_bootstrap.cc:492] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: No bootstrap required, opened a new log
I20260928 17:52:12.648967 8419 ts_tablet_manager.cc:1397] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:12.649096 8419 raft_consensus.cc:348] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.649179 8419 raft_consensus.cc:374] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:12.649211 8419 raft_consensus.cc:729] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Initialized, Role: FOLLOWER
I20260928 17:52:12.649261 8419 consensus_queue.cc:260] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.649346 8419 ts_tablet_manager.cc:1428] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:12.681787 8436 raft_consensus.cc:482] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:12.681895 8436 raft_consensus.cc:504] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.682090 8436 leader_election.cc:288] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001)
I20260928 17:52:12.682292 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4752a14ccdf14736a1aa9c3c0e9cd64d" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed" is_pre_election: true
I20260928 17:52:12.682351 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4752a14ccdf14736a1aa9c3c0e9cd64d" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34" is_pre_election: true
I20260928 17:52:12.682413 8064 raft_consensus.cc:2454] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:12.682431 8204 raft_consensus.cc:2454] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 0.
I20260928 17:52:12.682595 8276 leader_election.cc:302] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, eb66ff1dd84247758e80314c6cd204ed; no voters:
I20260928 17:52:12.682688 8436 raft_consensus.cc:2781] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:12.682729 8436 raft_consensus.cc:482] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:12.682749 8436 raft_consensus.cc:3037] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.683398 8436 raft_consensus.cc:504] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.683539 8436 leader_election.cc:288] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Requested vote from peers eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139), fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001)
I20260928 17:52:12.683704 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4752a14ccdf14736a1aa9c3c0e9cd64d" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa41f7c541354afe9ba9f923e87c2c34"
I20260928 17:52:12.683722 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4752a14ccdf14736a1aa9c3c0e9cd64d" candidate_uuid: "4b346bf7cbe44193b3358b1f62fecced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb66ff1dd84247758e80314c6cd204ed"
I20260928 17:52:12.683805 8204 raft_consensus.cc:3037] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.683817 8064 raft_consensus.cc:3037] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:12.684594 8064 raft_consensus.cc:2454] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:12.684594 8204 raft_consensus.cc:2454] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b346bf7cbe44193b3358b1f62fecced in term 1.
I20260928 17:52:12.684773 8276 leader_election.cc:302] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b346bf7cbe44193b3358b1f62fecced, eb66ff1dd84247758e80314c6cd204ed; no voters:
I20260928 17:52:12.684850 8436 raft_consensus.cc:2781] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:12.684898 8436 raft_consensus.cc:686] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 1 LEADER]: Becoming Leader. State: Replica: 4b346bf7cbe44193b3358b1f62fecced, State: Running, Role: LEADER
I20260928 17:52:12.684973 8436 consensus_queue.cc:237] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } }
I20260928 17:52:12.685572 7906 catalog_manager.cc:5495] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4b346bf7cbe44193b3358b1f62fecced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b346bf7cbe44193b3358b1f62fecced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34045 } health_report { overall_health: HEALTHY } } }
I20260928 17:52:12.700906 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.698438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.98.3:34045)
0928 17:52:12.698497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.698499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.698535 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:52:12.698586 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.698587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.698598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.698661 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.698744 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.699439 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.700213 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.700492 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.700568 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.700685 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.700716 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.700768 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.700814 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.700834 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:52:12.705883 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.702967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34045 (local address 127.0.98.3:34147)
0928 17:52:12.703122 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.703132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:12.703168 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:12.703288 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:12.703288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:12.703289 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 17:52:12.703327 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.703328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.704160 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.704161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.704866 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:12.704868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.705340 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.705341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:12.705440 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.705448 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.705651 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.705653 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:52:12.705753 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 17:52:12.705810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":4}
I20260928 17:52:12.705886 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.703107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34147 (local address 127.0.98.3:34045)
0928 17:52:12.703145 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:12.703147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:12.703175 (+ 28us) server_negotiation.cc:407] Connection header received
0928 17:52:12.703216 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:12.703217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:12.703226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:12.703284 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:12.703365 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.704032 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.704950 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:12.705254 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:12.705326 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:12.705537 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:12.705567 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:12.705622 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:12.705758 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 17:52:12.705783 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":22}
I20260928 17:52:12.707777 8204 raft_consensus.cc:1261] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.707923 8064 raft_consensus.cc:1261] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Refusing update from remote peer 4b346bf7cbe44193b3358b1f62fecced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:12.708029 8424 consensus_queue.cc:1035] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa41f7c541354afe9ba9f923e87c2c34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.708137 8428 consensus_queue.cc:1035] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb66ff1dd84247758e80314c6cd204ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:12.718680 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.718802 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:12.719432 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b70744b1247412b98cf8b62214166c5: Sending DeleteTablet for 3 replicas of tablet 4752a14ccdf14736a1aa9c3c0e9cd64d
I20260928 17:52:12.719656 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50862
I20260928 17:52:12.719751 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260928 17:52:12.719769 8476 tablet_replica.cc:330] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: stopping tablet replica
I20260928 17:52:12.719831 8476 raft_consensus.cc:2229] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.719874 8476 raft_consensus.cc:2258] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.719864 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39040
I20260928 17:52:12.719954 8477 tablet_replica.cc:330] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: stopping tablet replica
I20260928 17:52:12.720014 8477 raft_consensus.cc:2229] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:12.720052 8477 raft_consensus.cc:2258] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.720077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7979
I20260928 17:52:12.720213 8476 ts_tablet_manager.cc:1905] T 4752a14ccdf14736a1aa9c3c0e9cd64d P eb66ff1dd84247758e80314c6cd204ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.720439 8475 tablet_replica.cc:330] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: stopping tablet replica
I20260928 17:52:12.720503 8475 raft_consensus.cc:2229] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:12.720564 8475 raft_consensus.cc:2258] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:12.720947 8477 ts_tablet_manager.cc:1905] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.722675 8477 ts_tablet_manager.cc:1918] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:52:12.722715 8477 log.cc:1196] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/4752a14ccdf14736a1aa9c3c0e9cd64d
I20260928 17:52:12.722867 8477 ts_tablet_manager.cc:1939] T 4752a14ccdf14736a1aa9c3c0e9cd64d P fa41f7c541354afe9ba9f923e87c2c34: Deleting consensus metadata
I20260928 17:52:12.723354 8475 ts_tablet_manager.cc:1905] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:12.723361 7892 catalog_manager.cc:4841] TS fa41f7c541354afe9ba9f923e87c2c34 (127.0.98.2:34001): tablet 4752a14ccdf14736a1aa9c3c0e9cd64d (table test-table [id=efe3d7cf5ab046fe93157d206672cbc8]) successfully deleted
I20260928 17:52:12.724448 8475 ts_tablet_manager.cc:1918] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 17:52:12.724525 8475 log.cc:1196] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/4752a14ccdf14736a1aa9c3c0e9cd64d
I20260928 17:52:12.724711 8475 ts_tablet_manager.cc:1939] T 4752a14ccdf14736a1aa9c3c0e9cd64d P 4b346bf7cbe44193b3358b1f62fecced: Deleting consensus metadata
I20260928 17:52:12.724995 7892 catalog_manager.cc:4841] TS 4b346bf7cbe44193b3358b1f62fecced (127.0.98.3:34045): tablet 4752a14ccdf14736a1aa9c3c0e9cd64d (table test-table [id=efe3d7cf5ab046fe93157d206672cbc8]) successfully deleted
W20260928 17:52:12.726251 7892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46139 (error 108)
I20260928 17:52:12.726531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8119
I20260928 17:52:12.727176 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:12.726561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46139 (local address 127.0.0.1:50876)
0928 17:52:12.727078 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:12.727129 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":54,"threads_started":1}
W20260928 17:52:12.727710 7892 catalog_manager.cc:4584] TS eb66ff1dd84247758e80314c6cd204ed (127.0.98.1:46139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4752a14ccdf14736a1aa9c3c0e9cd64d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46139: connect: Connection refused (error 111)
I20260928 17:52:12.732501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8259
I20260928 17:52:12.738500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 7875
2026-09-28T17:52:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2756 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[8514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[8514](info): set up 2 sockets
Sep 28 17:52:12 dist-test-slave-zqlt krb5kdc[8514](info): commencing operation
krb5kdc: starting...
W20260928 17:52:13.776762 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:13 dist-test-slave-zqlt krb5kdc[8514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:52:13.892855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:13.973940 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:13.974172 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:13.974210 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:13.975376 8527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:13.975446 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:13.975474 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:13.975507 8527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:13.975539 8527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:13.977099 8527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:13.977366 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:13.977592 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:13.982276 8532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:13.982323 8533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:13.982313 8535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:13.982355 8527 server_base.cc:1034] running on GCE node
I20260928 17:52:13.982717 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:13.983031 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:13.984165 8527 hybrid_clock.cc:648] HybridClock initialized: now 1790617933984147 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T17:52:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1251 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:14 dist-test-slave-zqlt krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:14 dist-test-slave-zqlt krb5kdc[8540](info): set up 2 sockets
Sep 28 17:52:14 dist-test-slave-zqlt krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260928 17:52:15.036863 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:52:15.152973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:15.233184 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:15.233464 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:15.233503 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:15.234726 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:15.234802 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:15.234824 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:15.234846 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:15.234877 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:15.236549 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:15.236850 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:15.237092 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:15.242174 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.242448 8553 server_base.cc:1034] running on GCE node
W20260928 17:52:15.242178 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.242249 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.242858 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:15.243234 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:15.244398 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790617935244374 us; error 37 us; skew 500 ppm
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:15.251924 8553 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:52:15.252508 8553 webserver.cc:460] Webserver started at http://127.0.98.62:35377/ using document root <none> and password file <none>
I20260928 17:52:15.252748 8553 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:15.252795 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:15.252960 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:15.254102 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "74f562dda43f4fb8937f252f1559f3fa"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "dd57f1c2d65445a4f1e8d968c364ccd8"
server_key_iv: "ca03f783721659dcdf0c61605c5ea63b"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.254483 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "74f562dda43f4fb8937f252f1559f3fa"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "dd57f1c2d65445a4f1e8d968c364ccd8"
server_key_iv: "ca03f783721659dcdf0c61605c5ea63b"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.255837 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:15.256688 8568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.256871 8553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.256994 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "74f562dda43f4fb8937f252f1559f3fa"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "dd57f1c2d65445a4f1e8d968c364ccd8"
server_key_iv: "ca03f783721659dcdf0c61605c5ea63b"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.257067 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:15.275892 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:15.277820 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:15.277985 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:15.282109 8553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33327
I20260928 17:52:15.282142 8620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33327 every 8 connection(s)
I20260928 17:52:15.282651 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:15.283329 8621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:15.285754 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Bootstrap starting.
I20260928 17:52:15.286307 8621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:15.286542 8621 log.cc:824] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:15.287186 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: No bootstrap required, opened a new log
I20260928 17:52:15.287551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8553
I20260928 17:52:15.287680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:15.287878 392 external_mini_cluster.cc:1428] Setting key f77ddbe8fc7e6f8edbc2f342e94ee6f2
I20260928 17:52:15.289806 8621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:15.289944 8621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:15.289965 8621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f562dda43f4fb8937f252f1559f3fa, State: Initialized, Role: FOLLOWER
I20260928 17:52:15.290104 8621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:15.290189 8621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:15.290264 8621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:15.290329 8621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:15.291066 8621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:15.291153 8621 leader_election.cc:302] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f562dda43f4fb8937f252f1559f3fa; no voters:
I20260928 17:52:15.291342 8621 leader_election.cc:288] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:15.291371 8626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:15.291476 8626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 LEADER]: Becoming Leader. State: Replica: 74f562dda43f4fb8937f252f1559f3fa, State: Running, Role: LEADER
I20260928 17:52:15.291554 8626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:15.291604 8621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:15.291882 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f562dda43f4fb8937f252f1559f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } } }
I20260928 17:52:15.292037 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:15.292253 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f562dda43f4fb8937f252f1559f3fa. Latest consensus state: current_term: 1 leader_uuid: "74f562dda43f4fb8937f252f1559f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } } }
I20260928 17:52:15.292382 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:15.292817 8633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:15.293314 8633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:15.294943 8633 catalog_manager.cc:1347] Generated new cluster ID: 4cdc107d793c42d7b7fdfa2e42b5c011
I20260928 17:52:15.295002 8633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:15.302299 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.288418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.0.98.62:33327)
0928 17:52:15.288592 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:15.288602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:15.288635 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:52:15.289909 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:15.289918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:15.290009 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:15.290197 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:15.290969 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.292332 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.293446 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.293844 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.293957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.298375 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:15.298380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:15.298380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:15.298393 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:15.300580 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.300973 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.300976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.300977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.301123 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.301358 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.301359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.301359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.301715 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:15.301881 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:15.301956 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:52:15.302103 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260928 17:52:15.314073 8633 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:15.314601 8633 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:15.318427 8633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Generated new TSK 0
I20260928 17:52:15.318625 8633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:52:15.347316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:15.429929 8647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:15.430191 8647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:15.430233 8647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:15.431444 8647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:15.431515 8647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:15.431560 8647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:15.433110 8647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:15.433375 8647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:15.433609 8647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:15.438601 8655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.438608 8653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.438607 8652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.438870 8647 server_base.cc:1034] running on GCE node
I20260928 17:52:15.439102 8647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:15.439348 8647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:15.440519 8647 hybrid_clock.cc:648] HybridClock initialized: now 1790617935440509 us; error 29 us; skew 500 ppm
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:15.443197 8647 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:52:15.443703 8647 webserver.cc:460] Webserver started at http://127.0.98.1:38909/ using document root <none> and password file <none>
I20260928 17:52:15.443931 8647 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:15.443979 8647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:15.444175 8647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:15.445269 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a006c97d24d4481b77c256f01c7cc97"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "2678b7ed436eaa30f761c9abf5089381"
server_key_iv: "9475e9b56019489e7929b696c30f9924"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.445659 8647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a006c97d24d4481b77c256f01c7cc97"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "2678b7ed436eaa30f761c9abf5089381"
server_key_iv: "9475e9b56019489e7929b696c30f9924"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.447033 8647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:15.447762 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.447932 8647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:15.448030 8647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "7a006c97d24d4481b77c256f01c7cc97"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "2678b7ed436eaa30f761c9abf5089381"
server_key_iv: "9475e9b56019489e7929b696c30f9924"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.448136 8647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:15.461684 8647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:15.463948 8647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:15.464066 8647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:15.464366 8647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:15.464675 8647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:15.464707 8647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.464730 8647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:15.464771 8647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.470604 8647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44205
I20260928 17:52:15.470677 8775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44205 every 8 connection(s)
I20260928 17:52:15.471002 8647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:15.472435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8647
I20260928 17:52:15.472520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:15.472709 392 external_mini_cluster.cc:1428] Setting key 0c529dc76944801add4be381df22b9ab
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:52:15.481158 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.471743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50273 (local address 127.0.98.62:33327)
0928 17:52:15.471801 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:15.471803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:15.472023 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:52:15.473227 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:15.473229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:15.473240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:15.473287 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:15.474180 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.474754 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.475765 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.475980 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.476066 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.478349 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:15.478350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:15.478351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:15.478366 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:15.479732 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.480181 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.480183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.480183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.480265 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.480511 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.480512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.480512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.480625 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:15.480697 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:15.480978 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 17:52:15.481067 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:52:15.481343 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.471729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:50273)
0928 17:52:15.471899 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:15.471935 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:15.473126 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:15.473359 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:15.473363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:15.473779 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:15.474081 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.474087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.474864 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.474865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.475674 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.475676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.476073 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.476074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.476266 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.476827 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:15.476836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:15.478252 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:15.479814 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.479821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.479824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.480050 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.480326 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.480327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.480327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.480440 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.480688 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:15.480694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:15.480836 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 17:52:15.480945 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 17:52:15.481109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 17:52:15.481899 8776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33327
I20260928 17:52:15.482019 8776 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:15.482167 8776 heartbeater.cc:511] Master 127.0.98.62:33327 requested a full tablet report, sending...
I20260928 17:52:15.482653 8585 ts_manager.cc:194] Registered new tserver with Master: 7a006c97d24d4481b77c256f01c7cc97 (127.0.98.1:44205)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 17:52:15.483469 8585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:52:15.496779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:15.579038 8781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:15.579279 8781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:15.579319 8781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:15.580566 8781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:15.580627 8781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:15.580673 8781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:15.582219 8781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:15.582496 8781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:15.582726 8781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:15.587754 8786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.587805 8789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.587772 8787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.587965 8781 server_base.cc:1034] running on GCE node
I20260928 17:52:15.588325 8781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:15.588619 8781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:15.589811 8781 hybrid_clock.cc:648] HybridClock initialized: now 1790617935589788 us; error 34 us; skew 500 ppm
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:15.592540 8781 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:52:15.593003 8781 webserver.cc:460] Webserver started at http://127.0.98.2:35265/ using document root <none> and password file <none>
I20260928 17:52:15.593245 8781 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:15.593290 8781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:15.593461 8781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:15.594488 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd999e1cf10d46d3a5c3a6a48ea4cb00"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "e94427445c1d95cd994dd7ba3d1d47c6"
server_key_iv: "1760cfce9d385603cd0edbacf1992ec9"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.594842 8781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd999e1cf10d46d3a5c3a6a48ea4cb00"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "e94427445c1d95cd994dd7ba3d1d47c6"
server_key_iv: "1760cfce9d385603cd0edbacf1992ec9"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.596163 8781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:52:15.596844 8796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.597011 8781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:15.597090 8781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "dd999e1cf10d46d3a5c3a6a48ea4cb00"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "e94427445c1d95cd994dd7ba3d1d47c6"
server_key_iv: "1760cfce9d385603cd0edbacf1992ec9"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.597141 8781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:15.613559 8781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:15.616374 8781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:15.616544 8781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:15.616782 8781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:15.617103 8781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:15.617154 8781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.617197 8781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:15.617224 8781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.622933 8781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45057
I20260928 17:52:15.622962 8909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45057 every 8 connection(s)
I20260928 17:52:15.623435 8781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:15.631299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8781
I20260928 17:52:15.631428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:15.631640 392 external_mini_cluster.cc:1428] Setting key c36e0d6e7637bfe7b367fd9017376dec
I20260928 17:52:15.633457 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.624271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37127 (local address 127.0.98.62:33327)
0928 17:52:15.624319 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:15.624321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:15.624527 (+ 206us) server_negotiation.cc:407] Connection header received
0928 17:52:15.625721 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:15.625723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:15.625734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:15.625797 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:15.626406 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.626907 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.627903 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.628172 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.628238 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.630419 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:15.630420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:15.630421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:15.630429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:15.631999 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.632514 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.632515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.632516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.632573 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.632818 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.632819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.632820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.632934 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:15.633009 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:15.633280 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 17:52:15.633364 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:52:15.633685 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.624262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:37127)
0928 17:52:15.624397 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:15.624433 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:15.625598 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:15.625860 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:15.625863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:15.626131 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:15.626333 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.626338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.627015 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.627016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.627832 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.627834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.628246 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.628246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.628354 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.628918 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:15.628927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:15.630312 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:15.632124 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.632130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.632132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.632424 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.632641 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.632642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.632642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.632755 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.633013 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:15.633020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:15.633156 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 17:52:15.633262 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 17:52:15.633425 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260928 17:52:15.634239 8910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33327
I20260928 17:52:15.634349 8910 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:15.634526 8910 heartbeater.cc:511] Master 127.0.98.62:33327 requested a full tablet report, sending...
I20260928 17:52:15.634903 8585 ts_manager.cc:194] Registered new tserver with Master: dd999e1cf10d46d3a5c3a6a48ea4cb00 (127.0.98.2:45057)
I20260928 17:52:15.635438 8585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37127
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:52:15.654922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:15.737411 8915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:15.737627 8915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:15.737653 8915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:15.738737 8915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:15.738771 8915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:15.738796 8915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:15.740288 8915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:15.740494 8915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:15.740684 8915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:15.745597 8920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.745599 8923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.745713 8915 server_base.cc:1034] running on GCE node
W20260928 17:52:15.745599 8921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.746214 8915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:15.746464 8915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:15.747628 8915 hybrid_clock.cc:648] HybridClock initialized: now 1790617935747616 us; error 27 us; skew 500 ppm
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:15.750217 8915 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:52:15.750654 8915 webserver.cc:460] Webserver started at http://127.0.98.3:43141/ using document root <none> and password file <none>
I20260928 17:52:15.750839 8915 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:15.750878 8915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:15.750968 8915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:15.752010 8915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "61bd61b3cd8241cbb50ffed2a36e0a14"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "f014d44d0cf32f320c0997e3b6ffc88e"
server_key_iv: "ddee6ab611f85ab87a37602cbb2a2d01"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.752386 8915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61bd61b3cd8241cbb50ffed2a36e0a14"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "f014d44d0cf32f320c0997e3b6ffc88e"
server_key_iv: "ddee6ab611f85ab87a37602cbb2a2d01"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.753614 8915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:15.754236 8930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.754396 8915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:15.754478 8915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "61bd61b3cd8241cbb50ffed2a36e0a14"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "f014d44d0cf32f320c0997e3b6ffc88e"
server_key_iv: "ddee6ab611f85ab87a37602cbb2a2d01"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.754550 8915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:15.767597 8915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:15.769919 8915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:15.770076 8915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:15.770289 8915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:15.770579 8915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:15.770630 8915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.770671 8915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:15.770705 8915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.776183 8915 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39987
I20260928 17:52:15.776249 9043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39987 every 8 connection(s)
I20260928 17:52:15.776580 8915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:15.779796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 8915
I20260928 17:52:15.779923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:15.780167 392 external_mini_cluster.cc:1428] Setting key da3efe6726d905182623bdc99cd5e2a4
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:15.785852 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.777298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42269 (local address 127.0.98.62:33327)
0928 17:52:15.777361 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:15.777363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:15.777547 (+ 184us) server_negotiation.cc:407] Connection header received
0928 17:52:15.778463 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:15.778464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:15.778474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:15.778528 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:15.779190 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.779706 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.781139 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.781335 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.781389 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.783428 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:15.783429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:15.783429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:15.783436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:15.784672 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.785023 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.785024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.785024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.785073 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:15.785270 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:15.785270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:15.785271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:15.785365 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:15.785435 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:15.785730 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 17:52:15.785789 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:52:15.786003 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:15.777290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:42269)
0928 17:52:15.777434 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:15.777467 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:15.778365 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:15.778593 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:15.778597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:15.778882 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:15.779118 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.779124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.779792 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.779793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.781048 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:15.781050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:15.781398 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:15.781399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:15.781496 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:15.782016 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:15.782023 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:15.783347 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:15.784740 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.784746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.784747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.784959 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.785126 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:15.785127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:15.785127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:15.785220 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:15.785413 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:15.785417 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:15.785594 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 17:52:15.785691 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 17:52:15.785808 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 17:52:15.786470 9044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33327
I20260928 17:52:15.786577 9044 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:15.786753 9044 heartbeater.cc:511] Master 127.0.98.62:33327 requested a full tablet report, sending...
I20260928 17:52:15.787125 8585 ts_manager.cc:194] Registered new tserver with Master: 61bd61b3cd8241cbb50ffed2a36e0a14 (127.0.98.3:39987)
I20260928 17:52:15.787483 8585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42269
I20260928 17:52:15.791893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 17:52:15.858753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8647
I20260928 17:52:15.863421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8781
I20260928 17:52:15.867820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8915
I20260928 17:52:15.872364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 8553
I20260928 17:52:15.877465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33327
--webserver_interface=127.0.98.62
--webserver_port=35377
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:15.958588 9053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:15.958817 9053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:15.958841 9053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:15.960124 9053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:15.960171 9053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:15.960186 9053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:15.960199 9053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:15.960213 9053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:15.961798 9053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:15.962080 9053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:15.962339 9053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:15.967187 9061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.967172 9059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:15.967157 9058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:15.967412 9053 server_base.cc:1034] running on GCE node
I20260928 17:52:15.967664 9053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:15.967948 9053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:15.969105 9053 hybrid_clock.cc:648] HybridClock initialized: now 1790617935969101 us; error 53 us; skew 500 ppm
Sep 28 17:52:15 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:15.971589 9053 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:52:15.972002 9053 webserver.cc:460] Webserver started at http://127.0.98.62:35377/ using document root <none> and password file <none>
I20260928 17:52:15.972266 9053 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:15.972347 9053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:15.973739 9053 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:15.974409 9068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:15.974586 9053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:15.974679 9053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "74f562dda43f4fb8937f252f1559f3fa"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "dd57f1c2d65445a4f1e8d968c364ccd8"
server_key_iv: "ca03f783721659dcdf0c61605c5ea63b"
server_key_version: "encryptionkey@0"
I20260928 17:52:15.975024 9053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:15.991526 9053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:15.993810 9053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:15.993963 9053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:15.997767 9053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33327
I20260928 17:52:15.997807 9120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33327 every 8 connection(s)
I20260928 17:52:15.998369 9053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:15.999675 9121 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:52:16.000363 9121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Bootstrap starting.
I20260928 17:52:16.002249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9053
I20260928 17:52:16.002591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44205
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33327
--webserver_port=38909
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:52:16.008978 9121 log.cc:824] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:16.010684 9121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:52:16.010984 9121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Bootstrap complete.
I20260928 17:52:16.013628 9121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:16.013881 9121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f562dda43f4fb8937f252f1559f3fa, State: Initialized, Role: FOLLOWER
I20260928 17:52:16.014084 9121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:16.014176 9121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:16.014225 9121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:16.014276 9121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:52:16.015281 9121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:16.015400 9121 leader_election.cc:302] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f562dda43f4fb8937f252f1559f3fa; no voters:
I20260928 17:52:16.015585 9121 leader_election.cc:288] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:52:16.015650 9125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:52:16.015857 9121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:16.015887 9125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [term 2 LEADER]: Becoming Leader. State: Replica: 74f562dda43f4fb8937f252f1559f3fa, State: Running, Role: LEADER
I20260928 17:52:16.016196 9125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } }
I20260928 17:52:16.016528 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74f562dda43f4fb8937f252f1559f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } } }
I20260928 17:52:16.016625 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:16.016783 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f562dda43f4fb8937f252f1559f3fa. Latest consensus state: current_term: 2 leader_uuid: "74f562dda43f4fb8937f252f1559f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f562dda43f4fb8937f252f1559f3fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33327 } } }
I20260928 17:52:16.016832 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:16.016988 9134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:16.017586 9134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:16.017753 9134 catalog_manager.cc:1259] Loaded cluster ID: 4cdc107d793c42d7b7fdfa2e42b5c011
I20260928 17:52:16.017871 9134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:16.018888 9134 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:16.019093 9134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74f562dda43f4fb8937f252f1559f3fa: Loaded TSK: 0
I20260928 17:52:16.019315 9134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:52:16.103344 9123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:16.103569 9123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:16.103593 9123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:16.104839 9123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:16.104884 9123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:16.104912 9123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:16.106352 9123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38909
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:16.106591 9123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:16.106828 9123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:16.111855 9150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:16.111871 9147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:16.111871 9148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:16.112273 9123 server_base.cc:1034] running on GCE node
I20260928 17:52:16.112534 9123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:16.112824 9123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:16.114012 9123 hybrid_clock.cc:648] HybridClock initialized: now 1790617936113988 us; error 44 us; skew 500 ppm
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:16.117242 9123 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 17:52:16.117818 9123 webserver.cc:460] Webserver started at http://127.0.98.1:38909/ using document root <none> and password file <none>
I20260928 17:52:16.118077 9123 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:16.118160 9123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:16.119658 9123 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:16.120383 9157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.120564 9123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:16.120666 9123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "7a006c97d24d4481b77c256f01c7cc97"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "2678b7ed436eaa30f761c9abf5089381"
server_key_iv: "9475e9b56019489e7929b696c30f9924"
server_key_version: "encryptionkey@0"
I20260928 17:52:16.121078 9123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:16.139485 9123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:16.141871 9123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:16.142037 9123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:16.142284 9123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:16.142622 9123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:16.142673 9123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.142714 9123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:16.142740 9123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.149081 9123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44205
I20260928 17:52:16.149120 9270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44205 every 8 connection(s)
I20260928 17:52:16.149493 9123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.158181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9123
I20260928 17:52:16.158514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45057
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33327
--webserver_port=35265
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:52:16.159178 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.150243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:44115)
0928 17:52:16.150422 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.150470 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.150994 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.151494 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.151498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.152136 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.152375 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.152381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.154011 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.154012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.154832 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.154834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.155693 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.155694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.155805 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.156450 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.156459 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.158934 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260928 17:52:16.159534 9271 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33327: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260928 17:52:16.159849 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.150296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44115 (local address 127.0.98.62:33327)
0928 17:52:16.150524 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.150533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.150576 (+ 43us) server_negotiation.cc:407] Connection header received
0928 17:52:16.151205 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.151212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.151270 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.151423 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.152495 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.153906 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.154961 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.155589 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.155700 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.159103 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44115: BlockingRecv error: recv got EOF from 127.0.98.1:44115 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.165556 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.159779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:58177)
0928 17:52:16.159969 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.159983 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.160035 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.160273 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.160274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.160439 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.160499 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.160501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.161407 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.161408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.162312 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.162314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.162776 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.162777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.162862 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.163349 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.163355 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.165498 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260928 17:52:16.165699 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.159844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58177 (local address 127.0.98.62:33327)
0928 17:52:16.160152 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.160154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.160163 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:16.160187 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.160188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.160200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.160261 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.160580 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.161323 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.162399 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.162695 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.162762 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.165615 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58177: BlockingRecv error: recv got EOF from 127.0.98.1:58177 (error 108)
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":82,"threads_started":1}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.171113 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.166109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:44651)
0928 17:52:16.166147 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.166157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.166187 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.166335 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.166335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.166485 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.166517 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.166518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.167491 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.167492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.168291 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.168294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.168795 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.168796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.168834 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.169263 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.169268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.171044 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 17:52:16.171245 9271 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:52:16.171277 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.166165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44651 (local address 127.0.98.62:33327)
0928 17:52:16.166213 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.166215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.166224 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:16.166247 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.166248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.166258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.166324 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.166616 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.167401 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.168390 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.168720 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.168800 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.171137 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44651: BlockingRecv error: recv got EOF from 127.0.98.1:44651 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 17:52:16.242839 9275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:16.243100 9275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:16.243139 9275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:16.244572 9275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:16.244633 9275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:16.244670 9275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:16.246161 9275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35265
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:16.246363 9275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:16.246551 9275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:16.251511 9284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:16.251535 9281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:16.251777 9275 server_base.cc:1034] running on GCE node
W20260928 17:52:16.251539 9282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:16.252065 9275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:16.252338 9275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:16.253484 9275 hybrid_clock.cc:648] HybridClock initialized: now 1790617936253471 us; error 32 us; skew 500 ppm
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:16.255954 9275 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 17:52:16.256489 9275 webserver.cc:460] Webserver started at http://127.0.98.2:35265/ using document root <none> and password file <none>
I20260928 17:52:16.256717 9275 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:16.256788 9275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:16.258172 9275 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:16.258754 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.258921 9275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:16.259028 9275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "dd999e1cf10d46d3a5c3a6a48ea4cb00"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "e94427445c1d95cd994dd7ba3d1d47c6"
server_key_iv: "1760cfce9d385603cd0edbacf1992ec9"
server_key_version: "encryptionkey@0"
I20260928 17:52:16.259392 9275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:16.289374 9275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:16.291368 9275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:16.291548 9275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:16.291806 9275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:16.292173 9275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:16.292224 9275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.292268 9275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:16.292307 9275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.298331 9275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45057
I20260928 17:52:16.298380 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45057 every 8 connection(s)
I20260928 17:52:16.298743 9275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:16.304523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9275
I20260928 17:52:16.304857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39987
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33327
--webserver_port=43141
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:52:16.307226 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.299523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48971 (local address 127.0.98.62:33327)
0928 17:52:16.299812 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.299815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.299870 (+ 55us) server_negotiation.cc:407] Connection header received
0928 17:52:16.300576 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.300577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.300594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.300673 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.302051 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.302637 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.303644 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.303855 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.303946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.307169 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48971: BlockingRecv error: recv got EOF from 127.0.98.2:48971 (error 108)
Metrics: {"server-negotiator.queue_time_us":257}
I20260928 17:52:16.307291 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.299523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:48971)
0928 17:52:16.299698 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.299775 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.300466 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.300772 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.300776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.301640 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.301936 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.301943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.302738 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.302740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.303554 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.303556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.303949 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.303949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.304082 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.304629 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.304642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.307091 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
W20260928 17:52:16.307469 9405 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33327: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.312167 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.307725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:36651)
0928 17:52:16.307765 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.307774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.307809 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.307920 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.307921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.308051 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.308135 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.308137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.308847 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.308848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.309347 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.309348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.309690 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.309691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.309752 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.310178 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.310183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.312059 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 17:52:16.312234 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.307775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36651 (local address 127.0.98.62:33327)
0928 17:52:16.307819 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.307821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.307831 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:16.307858 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.307859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.307869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.307920 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.308242 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.308756 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.309429 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.309613 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.309681 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.312166 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36651: BlockingRecv error: recv got EOF from 127.0.98.2:36651 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.316695 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.312549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:55961)
0928 17:52:16.312593 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.312603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.312653 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.312784 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.312785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.312908 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.312935 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.312936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.313542 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.313543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.314035 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.314036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.314380 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.314381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.314430 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.314864 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.314869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.316613 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 17:52:16.316753 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.312626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55961 (local address 127.0.98.62:33327)
0928 17:52:16.312682 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.312684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.312692 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:16.312714 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.312715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.312724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.312780 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.313005 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.313467 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.314116 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.314313 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.314383 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.316686 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55961: BlockingRecv error: recv got EOF from 127.0.98.2:55961 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 17:52:16.317152 9405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:52:16.388231 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:16.388428 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:16.388451 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:16.389535 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:16.389571 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:16.389596 9408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:16.390928 9408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43141
--tserver_master_addrs=127.0.98.62:33327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:16.391104 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:16.391274 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:16.395912 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:16.396013 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:16.395946 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:16.396265 9408 server_base.cc:1034] running on GCE node
I20260928 17:52:16.396533 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:16.396762 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:16.397912 9408 hybrid_clock.cc:648] HybridClock initialized: now 1790617936397881 us; error 59 us; skew 500 ppm
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617936, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:16.400281 9408 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 17:52:16.400693 9408 webserver.cc:460] Webserver started at http://127.0.98.3:43141/ using document root <none> and password file <none>
I20260928 17:52:16.400907 9408 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:16.400983 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:16.402249 9408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:16.402752 9423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.402902 9408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:16.403004 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "61bd61b3cd8241cbb50ffed2a36e0a14"
format_stamp: "Formatted at 2026-09-28 17:52:15 on dist-test-slave-zqlt"
server_key: "f014d44d0cf32f320c0997e3b6ffc88e"
server_key_iv: "ddee6ab611f85ab87a37602cbb2a2d01"
server_key_version: "encryptionkey@0"
I20260928 17:52:16.403330 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:16.419780 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:16.421680 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:16.421842 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:16.422080 9408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:16.422369 9408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:16.422418 9408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.422456 9408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:16.422482 9408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:16.428678 9408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39987
I20260928 17:52:16.428727 9536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39987 every 8 connection(s)
I20260928 17:52:16.429064 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:16.429900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9408
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 17:52:16.438129 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.429794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44033 (local address 127.0.98.62:33327)
0928 17:52:16.429864 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.429866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.430088 (+ 222us) server_negotiation.cc:407] Connection header received
0928 17:52:16.430822 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.430823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.430835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.430902 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.431991 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.432728 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.433992 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.434280 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.434359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.438050 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44033: BlockingRecv error: recv got EOF from 127.0.98.3:44033 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:52:16.438184 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.429802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:44033)
0928 17:52:16.429959 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.429996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.430721 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.430962 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.430965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.431634 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.431900 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.431906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.432811 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.432812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.433899 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.433901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.434366 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.434367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.434470 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.434997 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.435006 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.437959 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
W20260928 17:52:16.438441 9537 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33327 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33327: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 17:52:16.438977 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.430421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.0.98.62:33327)
0928 17:52:16.430483 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.430485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.430590 (+ 105us) server_negotiation.cc:407] Connection header received
0928 17:52:16.430636 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.430637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.430651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.430728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.431367 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.432168 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.433139 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.433435 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.433515 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.434865 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:16.434869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:16.434870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:16.434881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:16.437133 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:16.437576 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:16.437580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:16.437581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:16.437709 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:16.437935 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:16.437958 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:16.437959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:16.438280 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:16.438415 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:16.438710 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 17:52:16.438842 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.443941 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.438646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:37113)
0928 17:52:16.438835 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.438846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.438880 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.439014 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.439015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.439164 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.439234 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.439235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.440141 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.440142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.441044 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.441046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.441480 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.441481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.441549 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.441958 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.441963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.443866 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
W20260928 17:52:16.444025 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.438806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37113 (local address 127.0.98.62:33327)
0928 17:52:16.438862 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.438864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.438886 (+ 22us) server_negotiation.cc:407] Connection header received
0928 17:52:16.438947 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.438948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.438956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.439025 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.439312 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.440024 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.441138 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.441404 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.441480 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.443963 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37113: BlockingRecv error: recv got EOF from 127.0.98.3:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:16 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:16.448381 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:16.444270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:55503)
0928 17:52:16.444344 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:16.444356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:16.444445 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:16.444665 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:16.444666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:16.444854 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:16.444904 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.444906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.445589 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.445590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.446123 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:16.446124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.446422 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.446423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:16.446462 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.446800 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:16.446804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:16.448326 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 17:52:16.448488 9274 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:16.444354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55503 (local address 127.0.98.62:33327)
0928 17:52:16.444391 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:16.444393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:16.444406 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:52:16.444557 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:16.444558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:16.444567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:16.444622 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:16.444981 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.445523 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.446189 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:16.446353 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:16.446399 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:16.448399 (+ 2000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55503: BlockingRecv error: recv got EOF from 127.0.98.3:55503 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:52:16.448503 9537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:17.176352 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:17.171676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:43017)
0928 17:52:17.171816 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:17.171828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:17.171891 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:17.172036 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:17.172037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:17.172215 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:17.172263 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.172265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.172928 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.172928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.173471 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.173472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.173797 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.173797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.173839 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.174353 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:17.174359 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:17.176277 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260928 17:52:17.176486 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:17.171745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43017 (local address 127.0.98.62:33327)
0928 17:52:17.171901 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:17.171903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:17.171912 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:17.171941 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:17.171941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:17.171952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:17.172019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:17.172330 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.172856 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.173542 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.173732 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.173783 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.176382 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43017: BlockingRecv error: recv got EOF from 127.0.98.1:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:17.322077 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:17.317553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:36321)
0928 17:52:17.317702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:17.317712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:17.317774 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:17.317935 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:17.317937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:17.318094 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:17.318144 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.318145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.318790 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.318791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.319361 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.319363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.319691 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.319692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.319733 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.320257 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:17.320261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:17.322007 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
W20260928 17:52:17.322238 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:17.317608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36321 (local address 127.0.98.62:33327)
0928 17:52:17.317661 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:17.317663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:17.317765 (+ 102us) server_negotiation.cc:407] Connection header received
0928 17:52:17.317813 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:17.317814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:17.317824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:17.317874 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:17.318215 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.318699 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.319437 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.319627 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.319679 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.322159 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36321: BlockingRecv error: recv got EOF from 127.0.98.2:36321 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:17 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:17.454053 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:17.448913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:47309)
0928 17:52:17.449103 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:17.449114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:17.449171 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:17.449356 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:17.449357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:17.449560 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:17.449635 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.449637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.450330 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.450331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.450908 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:17.450910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.451360 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.451361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:17.451429 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.451994 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:17.451999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:17.453977 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
W20260928 17:52:17.454185 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:17.449045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47309 (local address 127.0.98.62:33327)
0928 17:52:17.449103 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:17.449106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:17.449158 (+ 52us) server_negotiation.cc:407] Connection header received
0928 17:52:17.449218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:17.449219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:17.449231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:17.449291 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:17.449740 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.450230 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.451015 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:17.451212 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:17.451271 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:17.454084 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47309: BlockingRecv error: recv got EOF from 127.0.98.3:47309 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:18.181573 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:18.176800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:46331)
0928 17:52:18.176929 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:18.176942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:18.177002 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:18.177251 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:18.177252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:18.177430 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:18.177479 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.177481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.178209 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.178210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.178736 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.178737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.179081 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.179082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.179120 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.179648 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:18.179652 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:18.181466 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260928 17:52:18.181617 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:18.176974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46331 (local address 127.0.98.62:33327)
0928 17:52:18.177089 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:18.177091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:18.177098 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:18.177144 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:18.177144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:18.177157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:18.177234 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:18.177553 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.178141 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.178811 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.179003 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.179070 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.181543 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46331: BlockingRecv error: recv got EOF from 127.0.98.1:46331 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:18.327121 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:18.322541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:37295)
0928 17:52:18.322663 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:18.322674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:18.322738 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:18.322872 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:18.322873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:18.323028 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:18.323074 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.323075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.323750 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.323751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.324347 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.324349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.324678 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.324679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.324749 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.325209 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:18.325214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:18.327053 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
W20260928 17:52:18.327237 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:18.322605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37295 (local address 127.0.98.62:33327)
0928 17:52:18.322695 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:18.322697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:18.322740 (+ 43us) server_negotiation.cc:407] Connection header received
0928 17:52:18.322784 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:18.322785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:18.322797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:18.322867 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:18.323156 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.323663 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.324413 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.324609 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.324663 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.327146 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37295: BlockingRecv error: recv got EOF from 127.0.98.2:37295 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:18 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:18.459632 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:18.454600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:37347)
0928 17:52:18.454770 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:18.454786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:18.454859 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:18.455083 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:18.455084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:18.455260 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:18.455305 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.455307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.455984 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.455985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.456600 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:18.456602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.456972 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.456973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:18.457045 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.457547 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:18.457552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:18.459531 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
W20260928 17:52:18.459694 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:18.454865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37347 (local address 127.0.98.62:33327)
0928 17:52:18.454934 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:18.454936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:18.454944 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:18.454977 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:18.454978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:18.454990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:18.455063 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:18.455389 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.455893 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.456690 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:18.456887 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:18.456973 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:18.459624 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37347: BlockingRecv error: recv got EOF from 127.0.98.3:37347 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:19.186514 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:19.182035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:46519)
0928 17:52:19.182192 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:19.182203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:19.182278 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:19.182406 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:19.182407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:19.182568 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:19.182614 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.182616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.183231 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.183231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.183742 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.183744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.184071 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.184072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.184169 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.184610 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:19.184616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:19.186456 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260928 17:52:19.186662 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:19.182115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46519 (local address 127.0.98.62:33327)
0928 17:52:19.182213 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:19.182215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:19.182279 (+ 64us) server_negotiation.cc:407] Connection header received
0928 17:52:19.182332 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:19.182332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:19.182344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:19.182393 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:19.182675 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.183165 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.183815 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.184005 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.184061 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.186560 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46519: BlockingRecv error: recv got EOF from 127.0.98.1:46519 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:19.332057 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:19.327601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:42241)
0928 17:52:19.327746 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:19.327763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:19.327822 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:19.327973 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:19.327974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:19.328194 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:19.328242 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.328243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.328881 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.328881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.329408 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.329409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.329744 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.329745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.329784 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.330306 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:19.330311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:19.331990 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260928 17:52:19.332239 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:19.327691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42241 (local address 127.0.98.62:33327)
0928 17:52:19.327760 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:19.327762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:19.327821 (+ 59us) server_negotiation.cc:407] Connection header received
0928 17:52:19.327869 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:19.327870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:19.327882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:19.327958 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:19.328311 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.328805 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.329480 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.329672 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.329725 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.332148 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42241: BlockingRecv error: recv got EOF from 127.0.98.2:42241 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:19 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:19.465067 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:19.460198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:38533)
0928 17:52:19.460344 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:19.460358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:19.460446 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:19.460621 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:19.460623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:19.460787 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:19.460837 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.460838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.461482 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.461483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.462124 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:19.462127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.462484 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.462485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:19.462541 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.463106 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:19.463118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:19.465002 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
W20260928 17:52:19.465214 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:19.460322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38533 (local address 127.0.98.62:33327)
0928 17:52:19.460414 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:19.460416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:19.460427 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:19.460513 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:19.460514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:19.460526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:19.460569 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:19.460921 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.461411 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.462205 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:19.462394 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:19.462455 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:19.465119 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38533: BlockingRecv error: recv got EOF from 127.0.98.3:38533 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:20.191707 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:20.187088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:41273)
0928 17:52:20.187239 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:20.187249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:20.187299 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:20.187465 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:20.187465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:20.187639 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:20.187688 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.187689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.188448 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.188450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.188989 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.188990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.189342 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.189343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.189381 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.189922 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:20.189926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:20.191648 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260928 17:52:20.191918 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:20.187195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41273 (local address 127.0.98.62:33327)
0928 17:52:20.187332 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:20.187333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:20.187341 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:20.187366 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:20.187367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:20.187378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:20.187446 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:20.187759 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.188372 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.189080 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.189273 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.189329 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.191814 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41273: BlockingRecv error: recv got EOF from 127.0.98.1:41273 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:20.343055 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:20.332746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:54001)
0928 17:52:20.332892 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:20.332907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:20.332968 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:20.333109 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:20.333110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:20.333268 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:20.333317 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.333319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.333991 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.333992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.334556 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.334558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.334882 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.334883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.334924 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.341192 (+ 6268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:20.341197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:20.342991 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
W20260928 17:52:20.343333 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:20.332834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54001 (local address 127.0.98.62:33327)
0928 17:52:20.332911 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:20.332913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:20.332958 (+ 45us) server_negotiation.cc:407] Connection header received
0928 17:52:20.333015 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:20.333016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:20.333028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:20.333086 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:20.333412 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.333897 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.334636 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.334816 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.334865 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.343256 (+ 8391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54001: BlockingRecv error: recv got EOF from 127.0.98.2:54001 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:20 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:20.470613 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:20.465725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:52391)
0928 17:52:20.465897 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:20.465908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:20.465971 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:20.466118 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:20.466119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:20.466285 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:20.466334 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.466336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.467042 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.467043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.467871 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:20.467874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.468286 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.468287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:20.468327 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.468765 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:20.468769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:20.470552 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
W20260928 17:52:20.470818 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:20.465832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52391 (local address 127.0.98.62:33327)
0928 17:52:20.465909 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:20.465911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:20.465973 (+ 62us) server_negotiation.cc:407] Connection header received
0928 17:52:20.466019 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:20.466020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:20.466035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:20.466098 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:20.466402 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.466892 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.467976 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:20.468223 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:20.468280 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:20.470737 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52391: BlockingRecv error: recv got EOF from 127.0.98.3:52391 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:21.197154 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:21.192254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:49667)
0928 17:52:21.192414 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:21.192426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:21.192486 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:21.192662 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:21.192662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:21.192831 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:21.192885 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.192886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.193575 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.193577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.194137 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.194139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.194518 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.194518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.194558 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.195080 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:21.195085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:21.197085 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260928 17:52:21.197265 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:21.192344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49667 (local address 127.0.98.62:33327)
0928 17:52:21.192462 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:21.192464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:21.192479 (+ 15us) server_negotiation.cc:407] Connection header received
0928 17:52:21.192577 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:21.192578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:21.192588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:21.192641 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:21.192956 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.193475 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.194224 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.194449 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.194506 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.197184 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49667: BlockingRecv error: recv got EOF from 127.0.98.1:49667 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:21.348585 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:21.343749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:45705)
0928 17:52:21.343924 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:21.343936 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:21.344004 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:21.344193 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:21.344194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:21.344360 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:21.344413 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.344415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.345126 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.345126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.345656 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.345658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.346033 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.346034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.346073 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.346593 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:21.346598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:21.348440 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260928 17:52:21.348577 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:21.343922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45705 (local address 127.0.98.62:33327)
0928 17:52:21.343989 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:21.343990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:21.343999 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:21.344056 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:21.344057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:21.344069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:21.344184 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:21.344481 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.345051 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.345738 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.345964 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.346023 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.348522 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45705: BlockingRecv error: recv got EOF from 127.0.98.2:45705 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:21 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:21.475818 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:21.471113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:56439)
0928 17:52:21.471284 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:21.471299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:21.471386 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:21.471563 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:21.471564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:21.471734 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:21.471784 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.471785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.472547 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.472548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.473116 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:21.473118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.473440 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.473441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:21.473482 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.474028 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:21.474032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:21.475759 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
W20260928 17:52:21.475951 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:21.471238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56439 (local address 127.0.98.62:33327)
0928 17:52:21.471301 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:21.471303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:21.471378 (+ 75us) server_negotiation.cc:407] Connection header received
0928 17:52:21.471443 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:21.471444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:21.471455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:21.471509 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:21.471858 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.472457 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.473194 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:21.473375 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:21.473426 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:21.475850 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56439: BlockingRecv error: recv got EOF from 127.0.98.3:56439 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:22.202075 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:22.197691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:58037)
0928 17:52:22.197824 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:22.197835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:22.197892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:22.198025 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:22.198026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:22.198183 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:22.198232 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.198234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.198866 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.198867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.199383 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.199384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.199719 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.199720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.199782 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.200369 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:22.200373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:22.202018 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260928 17:52:22.202190 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:22.197733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58037 (local address 127.0.98.62:33327)
0928 17:52:22.197849 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:22.197851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:22.197891 (+ 40us) server_negotiation.cc:407] Connection header received
0928 17:52:22.197947 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:22.197948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:22.197959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:22.198007 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:22.198300 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.198799 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.199465 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.199653 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.199708 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.202109 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58037: BlockingRecv error: recv got EOF from 127.0.98.1:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:22.354099 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:22.349335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:54977)
0928 17:52:22.349488 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:22.349501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:22.349569 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:22.349784 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:22.349784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:22.350002 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:22.350061 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.350063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.350821 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.350822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.351343 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.351345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.351668 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.351669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.351709 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.352274 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:22.352280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:22.354039 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260928 17:52:22.354241 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:22.349400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54977 (local address 127.0.98.62:33327)
0928 17:52:22.349475 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:22.349477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:22.349577 (+ 100us) server_negotiation.cc:407] Connection header received
0928 17:52:22.349677 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:22.349677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:22.349707 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:22.349778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:22.350211 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.350743 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.351419 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.351600 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.351653 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.354161 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54977: BlockingRecv error: recv got EOF from 127.0.98.2:54977 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:22 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:22.481258 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:22.476344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:32943)
0928 17:52:22.476510 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:22.476522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:22.476587 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:22.476713 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:22.476714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:22.476876 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:22.476924 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.476926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.477608 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.477609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.478185 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:22.478187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.478534 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.478535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:22.478571 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.479204 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:22.479210 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:22.481197 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260928 17:52:22.481328 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:22.476422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32943 (local address 127.0.98.62:33327)
0928 17:52:22.476511 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:22.476518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:22.476586 (+ 68us) server_negotiation.cc:407] Connection header received
0928 17:52:22.476645 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:22.476646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:22.476657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:22.476697 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:22.476987 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.477530 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.478257 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:22.478445 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:22.478523 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:22.481278 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32943: BlockingRecv error: recv got EOF from 127.0.98.3:32943 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:23.207293 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:23.202545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:50257)
0928 17:52:23.202679 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:23.202697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:23.202746 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:23.202893 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:23.202894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:23.203049 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:23.203098 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.203099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.203741 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.203742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.204327 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.204329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.204663 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.204663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.204702 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.205313 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:23.205318 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:23.207222 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260928 17:52:23.207417 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:23.202648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50257 (local address 127.0.98.62:33327)
0928 17:52:23.202762 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:23.202764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:23.202772 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:23.202793 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:23.202794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:23.202804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:23.202866 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:23.203160 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.203664 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.204393 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.204598 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.204651 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.207311 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50257: BlockingRecv error: recv got EOF from 127.0.98.1:50257 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:23.359134 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:23.354654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:48981)
0928 17:52:23.354807 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:23.354818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:23.354873 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:23.355010 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:23.355011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:23.355173 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:23.355223 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.355225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.355850 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.355851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.356412 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.356414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.356747 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.356748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.356794 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.357268 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:23.357273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:23.359071 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260928 17:52:23.359256 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:23.354743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48981 (local address 127.0.98.62:33327)
0928 17:52:23.354824 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:23.354826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:23.354860 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:52:23.354925 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:23.354925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:23.354936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:23.354993 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:23.355286 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.355779 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.356476 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.356676 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.356728 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.359175 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48981: BlockingRecv error: recv got EOF from 127.0.98.2:48981 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:23 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:23.486601 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:23.481946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:56199)
0928 17:52:23.482103 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:23.482119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:23.482188 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:23.482326 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:23.482326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:23.482525 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:23.482571 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.482573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.483280 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.483281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.483811 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:23.483813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.484279 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.484280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:23.484349 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.484782 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:23.484786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:23.486502 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
W20260928 17:52:23.486639 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:23.482055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56199 (local address 127.0.98.62:33327)
0928 17:52:23.482146 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:23.482148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:23.482173 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:52:23.482248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:23.482250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:23.482262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:23.482305 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:23.482664 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.483180 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.483928 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:23.484191 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:23.484276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:23.486578 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56199: BlockingRecv error: recv got EOF from 127.0.98.3:56199 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:24.212790 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:24.207821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:56751)
0928 17:52:24.207963 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:24.207979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:24.208030 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:24.208271 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:24.208272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:24.208452 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:24.208503 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.208504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.209156 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.209158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.209692 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.209693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.210107 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.210108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.210148 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.210610 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:24.210615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:24.212666 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260928 17:52:24.212827 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:24.207984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56751 (local address 127.0.98.62:33327)
0928 17:52:24.208144 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:24.208146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:24.208153 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:52:24.208175 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:24.208175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:24.208186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:24.208251 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:24.208574 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.209067 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.209795 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.210026 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.210095 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.212755 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56751: BlockingRecv error: recv got EOF from 127.0.98.1:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:24.364713 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:24.359640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:55447)
0928 17:52:24.359755 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:24.359768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:24.359829 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:24.359959 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:24.359959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:24.360185 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:24.360232 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.360234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.361144 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.361144 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.361682 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.361684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.362229 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.362230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.362308 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.362913 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:24.362917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:24.364655 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
W20260928 17:52:24.364838 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:24.359714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55447 (local address 127.0.98.62:33327)
0928 17:52:24.359841 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:24.359843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:24.359851 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:24.359875 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:24.359876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:24.359886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:24.359936 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:24.360334 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.361068 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.361771 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.362123 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.362203 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.364750 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55447: BlockingRecv error: recv got EOF from 127.0.98.2:55447 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:24 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:24.491535 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:24.487126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:46859)
0928 17:52:24.487304 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:24.487316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:24.487392 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:24.487561 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:24.487562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:24.487726 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:24.487782 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.487783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.488428 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.488428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.488969 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:24.488970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.489271 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.489272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:24.489310 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.489747 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:24.489752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:24.491477 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
W20260928 17:52:24.491634 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:24.487182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46859 (local address 127.0.98.62:33327)
0928 17:52:24.487248 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:24.487251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:24.487395 (+ 144us) server_negotiation.cc:407] Connection header received
0928 17:52:24.487447 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:24.487448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:24.487463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:24.487537 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:24.487844 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.488366 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.489037 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:24.489209 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:24.489258 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:24.491568 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46859: BlockingRecv error: recv got EOF from 127.0.98.3:46859 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:25.218076 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:25.213409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:57945)
0928 17:52:25.213560 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:25.213570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:25.213626 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:25.213753 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:25.213754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:25.213920 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:25.213968 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.213970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.214617 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.214618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.215154 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.215156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.215623 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.215624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.215668 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.216117 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:25.216122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:25.218018 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
W20260928 17:52:25.218357 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:25.213506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57945 (local address 127.0.98.62:33327)
0928 17:52:25.213633 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:25.213635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:25.213645 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:25.213667 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:25.213668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:25.213678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:25.213738 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:25.214035 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.214542 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.215235 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.215538 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.215606 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.218245 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57945: BlockingRecv error: recv got EOF from 127.0.98.1:57945 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:25.373665 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:25.365229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:55363)
0928 17:52:25.365404 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:25.365415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:25.365504 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:25.365681 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:25.365682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:25.365865 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:25.365913 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.365915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.366660 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.366662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.367235 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.367237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.367622 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.367623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.367689 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.371645 (+ 3956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:25.371650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:25.373605 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
W20260928 17:52:25.373818 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:25.365304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55363 (local address 127.0.98.62:33327)
0928 17:52:25.365369 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:25.365372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:25.365484 (+ 112us) server_negotiation.cc:407] Connection header received
0928 17:52:25.365582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:25.365583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:25.365598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:25.365674 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:25.366011 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.366568 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.367320 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.367537 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.367613 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.373737 (+ 6124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55363: BlockingRecv error: recv got EOF from 127.0.98.2:55363 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:25 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:25.496615 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:25.492086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:39493)
0928 17:52:25.492246 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:25.492278 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:25.492340 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:25.492465 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:25.492466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:25.492637 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:25.492685 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.492686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.493332 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.493332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.493868 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:25.493870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.494187 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.494188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:25.494231 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.494672 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:25.494676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:25.496553 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
W20260928 17:52:25.496690 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:25.492168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39493 (local address 127.0.98.62:33327)
0928 17:52:25.492240 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:25.492242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:25.492347 (+ 105us) server_negotiation.cc:407] Connection header received
0928 17:52:25.492401 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:25.492402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:25.492416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:25.492459 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:25.492745 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.493256 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.493948 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:25.494121 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:25.494171 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:25.496641 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39493: BlockingRecv error: recv got EOF from 127.0.98.3:39493 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:26.223656 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:26.218619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:43693)
0928 17:52:26.218765 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:26.218777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:26.218831 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:26.219062 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:26.219064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:26.219270 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:26.219332 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.219340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.220054 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.220055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.220624 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.220626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.221026 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.221028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.221078 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.221526 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:26.221531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:26.223557 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260928 17:52:26.223707 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:26.218711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43693 (local address 127.0.98.62:33327)
0928 17:52:26.218853 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:26.218856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:26.218866 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:26.218892 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:26.218892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:26.218903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:26.218977 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:26.219446 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.219972 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.220725 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.220936 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.221001 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.223636 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43693: BlockingRecv error: recv got EOF from 127.0.98.1:43693 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:26.379004 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:26.374203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:48593)
0928 17:52:26.374342 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:26.374353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:26.374403 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:26.374590 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:26.374592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:26.374775 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:26.374825 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.374826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.375486 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.375486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.376002 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.376003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.376527 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.376528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.376599 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.377160 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:26.377165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:26.378940 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260928 17:52:26.379101 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:26.374318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48593 (local address 127.0.98.62:33327)
0928 17:52:26.374383 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:26.374385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:26.374403 (+ 18us) server_negotiation.cc:407] Connection header received
0928 17:52:26.374470 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:26.374470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:26.374481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:26.374524 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:26.374891 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.375404 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.376160 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.376439 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.376519 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.379034 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48593: BlockingRecv error: recv got EOF from 127.0.98.2:48593 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:26 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:26.501679 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:26.497154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:38983)
0928 17:52:26.497303 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:26.497315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:26.497377 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:26.497502 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:26.497503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:26.497669 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:26.497717 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.497719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.498352 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.498352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.498869 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:26.498870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.499189 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.499190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:26.499231 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.499758 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:26.499763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:26.501582 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
W20260928 17:52:26.501734 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:26.497277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38983 (local address 127.0.98.62:33327)
0928 17:52:26.497341 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:26.497342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:26.497363 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:52:26.497430 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:26.497431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:26.497442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:26.497482 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:26.497778 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.498287 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.498942 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:26.499123 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:26.499177 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:26.501667 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38983: BlockingRecv error: recv got EOF from 127.0.98.3:38983 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:27.234797 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:27.224174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:37727)
0928 17:52:27.224341 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:27.224359 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:27.224409 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:27.224565 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:27.224566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:27.224725 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:27.224774 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.224776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.225466 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.225467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.225981 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.225983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.226380 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.226380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.226417 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.232750 (+ 6333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:27.232756 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:27.234736 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
W20260928 17:52:27.234943 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:27.224263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37727 (local address 127.0.98.62:33327)
0928 17:52:27.224389 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:27.224391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:27.224402 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:52:27.224457 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:27.224457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:27.224470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:27.224547 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:27.224886 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.225388 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.226089 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.226305 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.226373 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.234853 (+ 8480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37727: BlockingRecv error: recv got EOF from 127.0.98.1:37727 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:27.384207 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:27.379582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:46449)
0928 17:52:27.379724 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:27.379734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:27.379795 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:27.379918 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:27.379919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:27.380103 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:27.380150 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.380152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.380844 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.380845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.381377 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.381379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.381839 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.381840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.381903 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.382362 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:27.382366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:27.384140 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260928 17:52:27.384263 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:27.379622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46449 (local address 127.0.98.62:33327)
0928 17:52:27.379688 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:27.379690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:27.379794 (+ 104us) server_negotiation.cc:407] Connection header received
0928 17:52:27.379841 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:27.379841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:27.379853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:27.379908 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:27.380244 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.380758 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.381483 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.381729 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.381817 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.384217 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46449: BlockingRecv error: recv got EOF from 127.0.98.2:46449 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:27 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:27.506661 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:27.502129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:59463)
0928 17:52:27.502244 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:27.502257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:27.502318 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:27.502449 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:27.502450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:27.502613 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:27.502660 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.502662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.503347 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.503348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.503916 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:27.503918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.504285 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.504286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:27.504352 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.504818 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:27.504822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:27.506603 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
W20260928 17:52:27.506794 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:27.502196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59463 (local address 127.0.98.62:33327)
0928 17:52:27.502293 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:27.502295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:27.502305 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:27.502381 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:27.502382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:27.502394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:27.502444 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:27.502717 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.503272 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.503994 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:27.504194 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:27.504271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:27.506717 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59463: BlockingRecv error: recv got EOF from 127.0.98.3:59463 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:28.240027 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:28.235403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:55021)
0928 17:52:28.235548 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:28.235558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:28.235610 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:28.235749 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:28.235750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:28.235917 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:28.235966 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.235967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.236615 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.236616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.237181 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.237183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.237545 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.237546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.237587 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.238128 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:28.238133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:28.239941 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20260928 17:52:28.240180 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:28.235493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55021 (local address 127.0.98.62:33327)
0928 17:52:28.235625 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:28.235627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:28.235636 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:28.235657 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:28.235657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:28.235668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:28.235731 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:28.236035 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.236549 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.237265 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.237476 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.237528 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.240056 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55021: BlockingRecv error: recv got EOF from 127.0.98.1:55021 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:28.389420 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:28.384705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:37055)
0928 17:52:28.384870 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:28.384890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:28.384947 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:28.385142 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:28.385143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:28.385377 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:28.385441 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.385443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.386111 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.386111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.386631 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.386633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.387007 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.387008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.387045 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.387549 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:28.387554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:28.389326 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
W20260928 17:52:28.389462 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:28.384813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37055 (local address 127.0.98.62:33327)
0928 17:52:28.384876 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:28.384878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:28.384955 (+ 77us) server_negotiation.cc:407] Connection header received
0928 17:52:28.385047 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:28.385048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:28.385061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:28.385127 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:28.385548 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.386044 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.386723 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.386935 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.387000 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.389401 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37055: BlockingRecv error: recv got EOF from 127.0.98.2:37055 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:28 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:28.511436 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:28.507078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:58529)
0928 17:52:28.507215 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:28.507225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:28.507283 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:28.507413 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:28.507414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:28.507592 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:28.507640 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.507641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.508301 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.508303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.508845 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:28.508847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.509170 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.509171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:28.509216 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.509657 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:28.509662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:28.511378 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260928 17:52:28.511598 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:28.507151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58529 (local address 127.0.98.62:33327)
0928 17:52:28.507196 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:28.507198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:28.507268 (+ 70us) server_negotiation.cc:407] Connection header received
0928 17:52:28.507338 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:28.507338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:28.507350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:28.507391 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:28.507702 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.508215 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.508919 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:28.509092 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:28.509143 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:28.511479 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58529: BlockingRecv error: recv got EOF from 127.0.98.3:58529 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:29.245337 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:29.240583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:57257)
0928 17:52:29.240742 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:29.240756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:29.240831 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:29.240999 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:29.240999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:29.241169 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:29.241222 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.241223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.241893 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.241894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.242412 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.242413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.242758 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.242758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.242802 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.243274 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:29.243279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:29.245241 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
W20260928 17:52:29.245412 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:29.240640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57257 (local address 127.0.98.62:33327)
0928 17:52:29.240818 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:29.240820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:29.240829 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:29.240923 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:29.240924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:29.240935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:29.240987 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:29.241285 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.241799 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.242498 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.242690 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.242743 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.245354 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57257: BlockingRecv error: recv got EOF from 127.0.98.1:57257 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:29.394321 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:29.389968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:42475)
0928 17:52:29.390133 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:29.390145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:29.390216 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:29.390363 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:29.390364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:29.390524 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:29.390574 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.390575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.391199 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.391200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.391718 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.391719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.392035 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.392035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.392073 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.392546 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:29.392550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:29.394251 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
W20260928 17:52:29.394457 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:29.390113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42475 (local address 127.0.98.62:33327)
0928 17:52:29.390177 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:29.390179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:29.390200 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:52:29.390277 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:29.390277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:29.390287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:29.390337 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:29.390637 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.391133 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.391790 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.391973 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.392024 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.394349 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42475: BlockingRecv error: recv got EOF from 127.0.98.2:42475 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:29 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:29.517086 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:29.511933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:60095)
0928 17:52:29.512079 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:29.512111 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:29.512175 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:29.512359 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:29.512360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:29.512549 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:29.512603 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.512605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.513279 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.513280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.513858 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:29.513860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.514190 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.514191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:29.514241 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.514757 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:29.514762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:29.516994 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260928 17:52:29.517179 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:29.512022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60095 (local address 127.0.98.62:33327)
0928 17:52:29.512116 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:29.512118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:29.512158 (+ 40us) server_negotiation.cc:407] Connection header received
0928 17:52:29.512233 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:29.512234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:29.512247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:29.512306 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:29.512685 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.513192 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.513936 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:29.514120 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:29.514170 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:29.517109 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60095: BlockingRecv error: recv got EOF from 127.0.98.3:60095 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:30.250535 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:30.245854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:59625)
0928 17:52:30.246015 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:30.246026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:30.246085 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:30.246220 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:30.246221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:30.246393 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:30.246441 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.246442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.247368 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.247369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.247923 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.247924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.248258 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.248258 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.248292 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.248720 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:30.248724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:30.250466 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
W20260928 17:52:30.250640 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:30.245966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59625 (local address 127.0.98.62:33327)
0928 17:52:30.246073 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:30.246075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:30.246084 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:30.246134 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:30.246134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:30.246144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:30.246201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:30.246526 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.247279 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.247993 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.248202 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.248258 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.250565 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59625: BlockingRecv error: recv got EOF from 127.0.98.1:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:30.402110 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:30.394809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:37709)
0928 17:52:30.394969 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:30.394980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:30.395047 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:30.395184 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:30.395185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:30.395341 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:30.395386 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.395388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.396017 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.396017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.396566 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.396568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.396922 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.396922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.396960 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.400194 (+ 3234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:30.400200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:30.402051 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
W20260928 17:52:30.402235 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:30.394911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37709 (local address 127.0.98.62:33327)
0928 17:52:30.394972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:30.394973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:30.395047 (+ 74us) server_negotiation.cc:407] Connection header received
0928 17:52:30.395094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:30.395095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:30.395106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:30.395164 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:30.395446 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.395951 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.396639 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.396851 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.396907 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.402151 (+ 5244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37709: BlockingRecv error: recv got EOF from 127.0.98.2:37709 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:30 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:30.522675 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:30.517803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.3:47555)
0928 17:52:30.517995 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:30.518009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:30.518099 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:30.518326 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:30.518327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:30.518516 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:30.518566 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.518568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.519261 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.519263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.519839 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:30.519840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.520302 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.520303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:30.520381 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.520819 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:30.520823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:30.522612 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
W20260928 17:52:30.522859 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:30.517861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47555 (local address 127.0.98.62:33327)
0928 17:52:30.517996 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:30.517999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:30.518076 (+ 77us) server_negotiation.cc:407] Connection header received
0928 17:52:30.518163 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:30.518165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:30.518180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:30.518281 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:30.518654 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.519170 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.519931 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:30.520205 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:30.520300 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:30.522742 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47555: BlockingRecv error: recv got EOF from 127.0.98.3:47555 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 17:52:31 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:31 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:31.255702 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.251086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.1:53287)
0928 17:52:31.251235 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:31.251248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:31.251328 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:31.251474 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:31.251475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:31.251636 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:31.251685 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.251687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.252442 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.252443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.252962 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.252964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.253351 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.253353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.253406 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.253910 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:31.253915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:31.255633 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260928 17:52:31.255795 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:31.251201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53287 (local address 127.0.98.62:33327)
0928 17:52:31.251348 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:31.251350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:31.251358 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:52:31.251379 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:31.251380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:31.251390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:31.251456 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:31.251749 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.252376 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.253031 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.253225 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.253276 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.255724 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53287: BlockingRecv error: recv got EOF from 127.0.98.1:53287 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
Sep 28 17:52:31 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 17:52:31 dist-test-slave-zqlt krb5kdc[8540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 17:52:31.407509 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.402634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33327 (local address 127.0.98.2:54105)
0928 17:52:31.402800 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:31.402811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:31.402873 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:31.403007 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:31.403008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:31.403173 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:31.403221 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.403223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.403913 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.403915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.404469 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.404471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.404844 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.404845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.404891 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.405439 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:31.405444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:31.407432 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33327: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
W20260928 17:52:31.407648 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:31.402741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54105 (local address 127.0.98.62:33327)
0928 17:52:31.402811 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:31.402813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:31.402858 (+ 45us) server_negotiation.cc:407] Connection header received
0928 17:52:31.402930 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:31.402931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:31.402941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:31.402986 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:31.403290 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.403818 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.404557 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.404768 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.404831 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.407566 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54105: BlockingRecv error: recv got EOF from 127.0.98.2:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 17:52:31.432394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9123
I20260928 17:52:31.436985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9275
I20260928 17:52:31.441255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9408
I20260928 17:52:31.445343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9053
2026-09-28T17:52:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17454 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T17:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:31Z Disabled control of system clock
I20260928 17:52:31.472611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 17:52:31.552268 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:31.552466 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:31.552485 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:31.553583 9604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:31.553617 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:31.553629 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:31.553642 9604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:31.553653 9604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:31.554925 9604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:31.555121 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:31.555347 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:31.560230 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:31.560424 9604 server_base.cc:1034] running on GCE node
W20260928 17:52:31.560276 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:31.560223 9609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:31.560761 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:31.561079 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:31.562266 9604 hybrid_clock.cc:648] HybridClock initialized: now 1790617951562244 us; error 37 us; skew 500 ppm
I20260928 17:52:31.563565 9604 webserver.cc:460] Webserver started at http://127.0.98.62:36433/ using document root <none> and password file <none>
I20260928 17:52:31.563815 9604 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:31.563884 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:31.564036 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:31.565119 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "f1efd7c6724149cc968faa40709fdac2"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "4faf48a75bb9ff8765bd3988642f30cc"
server_key_iv: "a1a28540c1e5f8415e9bde4beb2b4233"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.565469 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1efd7c6724149cc968faa40709fdac2"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "4faf48a75bb9ff8765bd3988642f30cc"
server_key_iv: "a1a28540c1e5f8415e9bde4beb2b4233"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.566744 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:31.567443 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.567621 9604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:31.567718 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "f1efd7c6724149cc968faa40709fdac2"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "4faf48a75bb9ff8765bd3988642f30cc"
server_key_iv: "a1a28540c1e5f8415e9bde4beb2b4233"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.567800 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:31.609946 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:31.610296 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:31.610456 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:31.614488 9604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42601
I20260928 17:52:31.614535 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42601 every 8 connection(s)
I20260928 17:52:31.614912 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:31.615600 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:31.617069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9604
I20260928 17:52:31.617197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:31.617398 392 external_mini_cluster.cc:1428] Setting key 6585628d7193d5ad4f9713a24e051ae6
I20260928 17:52:31.618846 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2: Bootstrap starting.
I20260928 17:52:31.619575 9671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:31.619863 9671 log.cc:824] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:31.620580 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2: No bootstrap required, opened a new log
I20260928 17:52:31.622344 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } }
I20260928 17:52:31.622509 9671 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:31.622556 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1efd7c6724149cc968faa40709fdac2, State: Initialized, Role: FOLLOWER
I20260928 17:52:31.622716 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } }
I20260928 17:52:31.622838 9671 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:31.622910 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:31.622991 9671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:31.623190 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.617976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.98.62:42601)
0928 17:52:31.618178 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:31.618187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:31.618212 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:52:31.618634 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:31.618640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:31.618699 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:31.618818 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:31.619064 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.620225 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.621090 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.621515 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.621656 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.622376 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:31.622380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:31.622380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:31.622586 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:31.622627 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:31.622642 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:31.622707 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:31.622803 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 17:52:31.622997 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260928 17:52:31.623704 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } }
I20260928 17:52:31.623812 9671 leader_election.cc:302] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1efd7c6724149cc968faa40709fdac2; no voters:
I20260928 17:52:31.623975 9671 leader_election.cc:288] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:31.624014 9676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:31.624182 9676 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [term 1 LEADER]: Becoming Leader. State: Replica: f1efd7c6724149cc968faa40709fdac2, State: Running, Role: LEADER
I20260928 17:52:31.624274 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:31.624330 9676 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } }
I20260928 17:52:31.624781 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1efd7c6724149cc968faa40709fdac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } } }
I20260928 17:52:31.624802 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1efd7c6724149cc968faa40709fdac2. Latest consensus state: current_term: 1 leader_uuid: "f1efd7c6724149cc968faa40709fdac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1efd7c6724149cc968faa40709fdac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42601 } } }
I20260928 17:52:31.624929 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:31.624933 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:31.625170 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:31.625676 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:31.627139 9685 catalog_manager.cc:1347] Generated new cluster ID: c1b2a0efe81a41bb850e02bbeb8e65d9
I20260928 17:52:31.627189 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:31.642894 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:31.643478 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:31.658056 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1efd7c6724149cc968faa40709fdac2: Generated new TSK 0
I20260928 17:52:31.658303 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:52:31.664974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 17:52:31.746901 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:31.747152 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:31.747187 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:31.748505 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:31.748570 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:31.748613 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:31.750058 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:31.750329 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:31.750633 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:31.755679 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:31.755800 9695 server_base.cc:1034] running on GCE node
W20260928 17:52:31.755661 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:31.755642 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:31.756189 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:31.756464 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:31.757671 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790617951757655 us; error 29 us; skew 500 ppm
I20260928 17:52:31.758960 9695 webserver.cc:460] Webserver started at http://127.0.98.1:36821/ using document root <none> and password file <none>
I20260928 17:52:31.759203 9695 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:31.759249 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:31.759440 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:31.760622 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc0eda0c27894f97868be8a11a962518"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "96dae9a3dd134648186b91b08e68ab42"
server_key_iv: "522b7404071ea607e952255d6a181199"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.760982 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc0eda0c27894f97868be8a11a962518"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "96dae9a3dd134648186b91b08e68ab42"
server_key_iv: "522b7404071ea607e952255d6a181199"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.762313 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:31.763108 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.763320 9695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:31.763410 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "cc0eda0c27894f97868be8a11a962518"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "96dae9a3dd134648186b91b08e68ab42"
server_key_iv: "522b7404071ea607e952255d6a181199"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.763486 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:31.778961 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:31.779325 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:31.779486 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:31.779726 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:31.780186 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:31.780248 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.780283 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:31.780318 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.786398 9695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41663
I20260928 17:52:31.786437 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41663 every 8 connection(s)
I20260928 17:52:31.786798 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:31.790325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9695
I20260928 17:52:31.790411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:31.790655 392 external_mini_cluster.cc:1428] Setting key bcf0c389f7396c623241bb9aa4428168
I20260928 17:52:31.791988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 17:52:31.792924 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.787708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49099 (local address 127.0.98.62:42601)
0928 17:52:31.787779 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:31.787781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:31.788117 (+ 336us) server_negotiation.cc:407] Connection header received
0928 17:52:31.788681 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:31.788683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:31.788696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:31.788750 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:31.789477 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.790008 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.791244 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.791499 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.791577 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.792671 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:31.792673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:31.792673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:31.792678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:31.792692 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:31.792696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:31.792739 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:31.792837 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:52:31.792861 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:52:31.793306 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.787759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42601 (local address 127.0.98.1:49099)
0928 17:52:31.787956 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:31.787992 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:31.788567 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:31.788843 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:31.788847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:31.788856 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:31.789376 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.789380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.790121 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.790122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.791158 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.791161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.791624 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.791626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.791825 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.792482 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:31.792483 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:31.792532 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:31.792570 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:31.792746 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:31.792750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:31.792753 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:31.792835 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 17:52:31.792981 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 17:52:31.794013 9823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42601
I20260928 17:52:31.794130 9823 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:31.794348 9823 heartbeater.cc:511] Master 127.0.98.62:42601 requested a full tablet report, sending...
I20260928 17:52:31.794946 9635 ts_manager.cc:194] Registered new tserver with Master: cc0eda0c27894f97868be8a11a962518 (127.0.98.1:41663)
I20260928 17:52:31.795686 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49099
W20260928 17:52:31.874857 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:31.875094 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:31.875130 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:31.876360 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:31.876418 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:31.876461 9826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:31.877892 9826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:31.878155 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:31.878454 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:31.883311 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:31.883457 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:31.883459 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:31.883781 9826 server_base.cc:1034] running on GCE node
I20260928 17:52:31.883967 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:31.884253 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:31.885402 9826 hybrid_clock.cc:648] HybridClock initialized: now 1790617951885386 us; error 34 us; skew 500 ppm
I20260928 17:52:31.886695 9826 webserver.cc:460] Webserver started at http://127.0.98.2:41619/ using document root <none> and password file <none>
I20260928 17:52:31.886942 9826 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:31.887009 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:31.887146 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:31.888338 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "db852b196acf43f8aa845042527252d5"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "0e1e100b155dc6345a27b0d67ffefeee"
server_key_iv: "8ada0d85ef30be06be5ad3022b35d3c7"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.888706 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db852b196acf43f8aa845042527252d5"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "0e1e100b155dc6345a27b0d67ffefeee"
server_key_iv: "8ada0d85ef30be06be5ad3022b35d3c7"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.890012 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:31.890705 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.890910 9826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:31.890993 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "db852b196acf43f8aa845042527252d5"
format_stamp: "Formatted at 2026-09-28 17:52:31 on dist-test-slave-zqlt"
server_key: "0e1e100b155dc6345a27b0d67ffefeee"
server_key_iv: "8ada0d85ef30be06be5ad3022b35d3c7"
server_key_version: "encryptionkey@0"
I20260928 17:52:31.891045 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:31.911589 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:31.911877 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:31.911979 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:31.912241 9826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:31.912559 9826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:31.912590 9826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.912611 9826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:31.912624 9826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:31.918072 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37929
I20260928 17:52:31.918151 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37929 every 8 connection(s)
I20260928 17:52:31.918509 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:31.924054 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.919336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52089 (local address 127.0.98.62:42601)
0928 17:52:31.919398 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:31.919400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:31.919630 (+ 230us) server_negotiation.cc:407] Connection header received
0928 17:52:31.920291 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:31.920292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:31.920302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:31.920366 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:31.921048 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.921548 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.922578 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.922802 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.922868 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.923723 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:31.923724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:31.923724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:31.923755 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:31.923771 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:31.923774 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:31.923823 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:31.923957 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 17:52:31.923983 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:52:31.924243 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:31.919288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42601 (local address 127.0.98.2:52089)
0928 17:52:31.919482 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:31.919525 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:31.920182 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:31.920425 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:31.920428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:31.920437 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:31.920952 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.920956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.921654 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.921663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.922474 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:31.922476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:31.922916 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:31.922917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:31.923051 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:31.923574 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:31.923574 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:31.923602 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:31.923628 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:31.923849 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:31.923853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:31.923855 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:52:31.923911 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:52:31.923995 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260928 17:52:31.924758 9954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42601
I20260928 17:52:31.924872 9954 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:31.925093 9954 heartbeater.cc:511] Master 127.0.98.62:42601 requested a full tablet report, sending...
I20260928 17:52:31.925511 9635 ts_manager.cc:194] Registered new tserver with Master: db852b196acf43f8aa845042527252d5 (127.0.98.2:37929)
I20260928 17:52:31.925895 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52089
I20260928 17:52:31.928282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9826
I20260928 17:52:31.928354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:31.928503 392 external_mini_cluster.cc:1428] Setting key 24343a213f77ec1e700d9afc55d4d4c4
I20260928 17:52:31.929404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--builtin_ntp_servers=127.0.98.20:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 17:52:32.010802 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:32.011016 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:32.011036 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:32.012252 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:32.012293 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:32.012319 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:32.013648 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:32.013921 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:32.014215 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:32.019418 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.019418 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.019510 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.019667 9957 server_base.cc:1034] running on GCE node
I20260928 17:52:32.020054 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:32.020388 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:32.021565 9957 hybrid_clock.cc:648] HybridClock initialized: now 1790617952021551 us; error 29 us; skew 500 ppm
I20260928 17:52:32.023030 9957 webserver.cc:460] Webserver started at http://127.0.98.3:34961/ using document root <none> and password file <none>
I20260928 17:52:32.023298 9957 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:32.023372 9957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:32.023548 9957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:32.024933 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "c494eb42dbb44e8a825e0a396f899c41"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "983839706444bbba868877e3e0aa42c4"
server_key_iv: "ea00a5d9ab1ad2c7e8b93dd94546bbf6"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.025318 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c494eb42dbb44e8a825e0a396f899c41"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "983839706444bbba868877e3e0aa42c4"
server_key_iv: "ea00a5d9ab1ad2c7e8b93dd94546bbf6"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.026716 9957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:32.027570 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.027807 9957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:32.027880 9957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "c494eb42dbb44e8a825e0a396f899c41"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "983839706444bbba868877e3e0aa42c4"
server_key_iv: "ea00a5d9ab1ad2c7e8b93dd94546bbf6"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.027976 9957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:32.055645 9957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:32.056021 9957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:32.056216 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:32.056483 9957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:32.056871 9957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:32.056931 9957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.056967 9957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:32.056996 9957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.062773 9957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33073
I20260928 17:52:32.062815 10084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33073 every 8 connection(s)
I20260928 17:52:32.063213 9957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:32.064441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 9957
I20260928 17:52:32.064546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:32.064772 392 external_mini_cluster.cc:1428] Setting key b212135a4e6e9190aca25dc9ca8068ee
I20260928 17:52:32.069351 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.064150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36821 (local address 127.0.98.62:42601)
0928 17:52:32.064228 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.064251 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.064587 (+ 336us) server_negotiation.cc:407] Connection header received
0928 17:52:32.065306 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.065307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.065317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.065365 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.066223 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:32.066744 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:32.067899 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:32.068115 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:32.068196 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:32.069059 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:32.069060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:32.069061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:32.069066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:32.069088 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:32.069092 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:32.069151 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:32.069245 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 17:52:32.069273 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:52:32.069614 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.064145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42601 (local address 127.0.98.3:36821)
0928 17:52:32.064428 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:32.064479 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:32.065205 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:32.065443 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:32.065447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:32.065458 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:32.066115 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:32.066120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:32.066842 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:32.066843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:32.067792 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:32.067795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:32.068204 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:32.068206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:32.068351 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:32.068908 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:32.068908 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:32.068937 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:32.068961 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:32.069152 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:32.069158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:32.069161 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:32.069250 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:52:32.069373 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":134,"threads_started":1}
I20260928 17:52:32.070153 10085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42601
I20260928 17:52:32.070236 10085 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:32.070431 10085 heartbeater.cc:511] Master 127.0.98.62:42601 requested a full tablet report, sending...
I20260928 17:52:32.070879 9635 ts_manager.cc:194] Registered new tserver with Master: c494eb42dbb44e8a825e0a396f899c41 (127.0.98.3:33073)
I20260928 17:52:32.071280 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36821
I20260928 17:52:32.076943 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:52:32.079267 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:32.078650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.62:42601)
0928 17:52:32.078729 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.078731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.078842 (+ 111us) server_negotiation.cc:407] Connection header received
0928 17:52:32.078929 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.078930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.078942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.079001 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.079147 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49248: BlockingRecv error: recv got EOF from 127.0.0.1:49248 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 17:52:32.079315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9695
I20260928 17:52:32.084082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9826
I20260928 17:52:32.088191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9957
I20260928 17:52:32.092451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 9604
2026-09-28T17:52:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (646 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T17:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:32Z Disabled control of system clock
I20260928 17:52:32.118801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40071
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:52:32.198969 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:32.199208 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:32.199244 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:32.200408 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:32.200451 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:32.200464 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:32.200477 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:32.200489 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:32.201829 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:32.202036 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:32.202288 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:32.207297 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.207309 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.207549 10096 server_base.cc:1034] running on GCE node
W20260928 17:52:32.207618 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.207861 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:32.208243 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:32.209419 10096 hybrid_clock.cc:648] HybridClock initialized: now 1790617952209398 us; error 43 us; skew 500 ppm
I20260928 17:52:32.210722 10096 webserver.cc:460] Webserver started at http://127.0.98.62:35585/ using document root <none> and password file <none>
I20260928 17:52:32.210968 10096 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:32.211037 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:32.211205 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:32.212450 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "dc4de70ea1c84825a1adb4696859c863"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "02e3ebdda22d5189f4a8cc3cd2643672"
server_key_iv: "31a1245b874d6c2f5e185134a95be582"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.212885 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc4de70ea1c84825a1adb4696859c863"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "02e3ebdda22d5189f4a8cc3cd2643672"
server_key_iv: "31a1245b874d6c2f5e185134a95be582"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.214241 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:32.215058 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.215232 10096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:32.215355 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "dc4de70ea1c84825a1adb4696859c863"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "02e3ebdda22d5189f4a8cc3cd2643672"
server_key_iv: "31a1245b874d6c2f5e185134a95be582"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.215433 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:32.218976 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:32.219270 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:32.219421 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:32.223613 10096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40071
I20260928 17:52:32.223645 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40071 every 8 connection(s)
I20260928 17:52:32.224000 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:32.224727 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:32.227561 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: Bootstrap starting.
I20260928 17:52:32.228212 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:32.228463 10163 log.cc:824] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:32.229103 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: No bootstrap required, opened a new log
I20260928 17:52:32.230651 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } }
I20260928 17:52:32.230808 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:32.230832 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4de70ea1c84825a1adb4696859c863, State: Initialized, Role: FOLLOWER
I20260928 17:52:32.230981 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } }
I20260928 17:52:32.231061 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:32.231086 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:32.231112 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:32.231631 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } }
I20260928 17:52:32.231714 10163 leader_election.cc:302] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc4de70ea1c84825a1adb4696859c863; no voters:
I20260928 17:52:32.231886 10163 leader_election.cc:288] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:32.231920 10166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:32.232038 10166 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [term 1 LEADER]: Becoming Leader. State: Replica: dc4de70ea1c84825a1adb4696859c863, State: Running, Role: LEADER
I20260928 17:52:32.232196 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:32.232231 10166 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } }
I20260928 17:52:32.232645 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc4de70ea1c84825a1adb4696859c863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } } }
I20260928 17:52:32.232975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10096
I20260928 17:52:32.233081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:32.233321 392 external_mini_cluster.cc:1428] Setting key 28c9c1f788077ba3de82e616f84e1c58
I20260928 17:52:32.232940 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4de70ea1c84825a1adb4696859c863. Latest consensus state: current_term: 1 leader_uuid: "dc4de70ea1c84825a1adb4696859c863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4de70ea1c84825a1adb4696859c863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40071 } } }
I20260928 17:52:32.233464 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:32.232749 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863 [sys.catalog]: This master's current role is: LEADER
W20260928 17:52:32.234424 10182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:52:32.234483 10182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:52:32.234529 10181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:32.235071 10181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:32.235747 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.233863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.98.62:40071)
0928 17:52:32.234054 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.234067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.234105 (+ 38us) server_negotiation.cc:407] Connection header received
0928 17:52:32.234220 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.234226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.234290 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.234456 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.235230 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:32.235234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:32.235235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:32.235240 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:32.235299 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:32.235314 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:32.235393 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:32.235483 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 17:52:32.235548 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260928 17:52:32.239960 10181 catalog_manager.cc:1347] Generated new cluster ID: 2ee2ff10d40340878f55627dadb388a9
I20260928 17:52:32.240046 10181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:52:32.257112 10181 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:32.257350 10181 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:32.271070 10181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc4de70ea1c84825a1adb4696859c863: Generated new TSK 0
I20260928 17:52:32.271340 10181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:52:32.281652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:52:32.365165 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:32.365396 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:32.365438 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:32.366712 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:32.366771 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:32.366822 10187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:32.368290 10187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:32.368551 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:32.368852 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:52:32.373948 10187 server_base.cc:1034] running on GCE node
W20260928 17:52:32.373951 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.373904 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.373909 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.374262 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:32.374516 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:32.375697 10187 hybrid_clock.cc:648] HybridClock initialized: now 1790617952375695 us; error 41 us; skew 500 ppm
I20260928 17:52:32.377115 10187 webserver.cc:460] Webserver started at http://127.0.98.1:42869/ using document root <none> and password file <none>
I20260928 17:52:32.377388 10187 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:32.377461 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:32.377614 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:32.378692 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "c94d1b39ea6a435f87f3c0977359e1c6"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "a6ad62dce4f7c8bb5de4ff071d7e625f"
server_key_iv: "0b36da5a7650478b401718fd3ff53061"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.379070 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c94d1b39ea6a435f87f3c0977359e1c6"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "a6ad62dce4f7c8bb5de4ff071d7e625f"
server_key_iv: "0b36da5a7650478b401718fd3ff53061"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.380429 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:52:32.381204 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.381378 10187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:52:32.381484 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "c94d1b39ea6a435f87f3c0977359e1c6"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "a6ad62dce4f7c8bb5de4ff071d7e625f"
server_key_iv: "0b36da5a7650478b401718fd3ff53061"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.381557 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:32.384972 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:32.385237 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:32.385380 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:32.385586 10187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:32.385885 10187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:32.385938 10187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.385977 10187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:32.386004 10187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.391803 10187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40389
I20260928 17:52:32.391841 10314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40389 every 8 connection(s)
I20260928 17:52:32.392220 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:52:32.395404 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.393015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37147 (local address 127.0.98.62:40071)
0928 17:52:32.393060 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.393061 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.393313 (+ 252us) server_negotiation.cc:407] Connection header received
0928 17:52:32.393827 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.393828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.393836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.393902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.395123 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:32.395124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:32.395124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:32.395130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:32.395148 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:32.395153 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:32.395213 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:32.395315 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 17:52:32.395334 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:52:32.395623 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.392996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40071 (local address 127.0.98.1:37147)
0928 17:52:32.393192 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:32.393225 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:32.393714 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:32.393990 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:32.393993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:32.394002 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:32.394961 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:32.394962 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:32.394990 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:32.395020 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:32.395214 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:32.395219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:32.395222 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:32.395313 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 17:52:32.395383 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260928 17:52:32.396211 10315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40071
I20260928 17:52:32.396291 10315 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:32.396406 10315 heartbeater.cc:511] Master 127.0.98.62:40071 requested a full tablet report, sending...
I20260928 17:52:32.396796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10187
I20260928 17:52:32.396875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:32.396963 10127 ts_manager.cc:194] Registered new tserver with Master: c94d1b39ea6a435f87f3c0977359e1c6 (127.0.98.1:40389)
I20260928 17:52:32.397115 392 external_mini_cluster.cc:1428] Setting key 8c8748f6cedde29177ced52d37544875
I20260928 17:52:32.398033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:52:32.484340 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:32.484634 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:32.484665 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:32.485986 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:32.486028 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:32.486056 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:32.487432 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:32.487702 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:32.488029 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:32.493083 10324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.493093 10323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.493104 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.493423 10318 server_base.cc:1034] running on GCE node
I20260928 17:52:32.493631 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:32.493904 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:32.495102 10318 hybrid_clock.cc:648] HybridClock initialized: now 1790617952495092 us; error 38 us; skew 500 ppm
I20260928 17:52:32.496618 10318 webserver.cc:460] Webserver started at http://127.0.98.2:33173/ using document root <none> and password file <none>
I20260928 17:52:32.496879 10318 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:32.496955 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:32.497119 10318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:32.498298 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "31b20085aedd45e0b488fde38b5dedad"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "97306a85946bbd5e3a5b67ec270926fc"
server_key_iv: "5002cbb350e478bdade487ef0df84e32"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.499300 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31b20085aedd45e0b488fde38b5dedad"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "97306a85946bbd5e3a5b67ec270926fc"
server_key_iv: "5002cbb350e478bdade487ef0df84e32"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.500946 10318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:32.501791 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.502095 10318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:52:32.502210 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "31b20085aedd45e0b488fde38b5dedad"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "97306a85946bbd5e3a5b67ec270926fc"
server_key_iv: "5002cbb350e478bdade487ef0df84e32"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.502296 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:32.506028 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:32.506322 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:32.506474 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:32.506709 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:32.507041 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:32.507099 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.507143 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:32.507172 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.513484 10318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37005
I20260928 17:52:32.513521 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37005 every 8 connection(s)
I20260928 17:52:32.513897 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:32.517089 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.514725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33513 (local address 127.0.98.62:40071)
0928 17:52:32.514802 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.514805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.515077 (+ 272us) server_negotiation.cc:407] Connection header received
0928 17:52:32.515560 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.515561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.515567 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.515629 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.516859 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:32.516861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:32.516861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:32.516865 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:32.516883 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:32.516888 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:32.516945 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:32.517018 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:52:32.517032 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:52:32.517271 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.514707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40071 (local address 127.0.98.2:33513)
0928 17:52:32.514946 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:32.514982 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:32.515466 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:32.515696 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:32.515700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:32.515708 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:32.516704 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:32.516705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:32.516733 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:32.516763 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:32.516951 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:32.516954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:32.516956 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:52:32.517012 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:52:32.517061 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":120,"threads_started":1}
I20260928 17:52:32.517840 10446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40071
I20260928 17:52:32.517939 10446 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:32.518086 10446 heartbeater.cc:511] Master 127.0.98.62:40071 requested a full tablet report, sending...
I20260928 17:52:32.518448 10127 ts_manager.cc:194] Registered new tserver with Master: 31b20085aedd45e0b488fde38b5dedad (127.0.98.2:37005)
I20260928 17:52:32.522821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10318
I20260928 17:52:32.522939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:32.523157 392 external_mini_cluster.cc:1428] Setting key bd1a40afbe41977410714dc60d230cd6
I20260928 17:52:32.524290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--builtin_ntp_servers=127.0.98.20:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 17:52:32.607589 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:32.607803 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:32.607826 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:32.609176 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:32.609223 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:32.609252 10449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:32.610699 10449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:32.610996 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:32.611377 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:32.616567 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.616542 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:32.616559 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:32.616716 10449 server_base.cc:1034] running on GCE node
I20260928 17:52:32.617000 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:32.617270 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:32.618469 10449 hybrid_clock.cc:648] HybridClock initialized: now 1790617952618456 us; error 30 us; skew 500 ppm
I20260928 17:52:32.619947 10449 webserver.cc:460] Webserver started at http://127.0.98.3:37201/ using document root <none> and password file <none>
I20260928 17:52:32.620265 10449 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:32.620342 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:32.620507 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:32.621717 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf00cbc4ecad4a59a01874f0734e6567"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "8172f6677ed014be479d80188fca511f"
server_key_iv: "15d388b93ee707a9c89986cdeea2aeef"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.622102 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf00cbc4ecad4a59a01874f0734e6567"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "8172f6677ed014be479d80188fca511f"
server_key_iv: "15d388b93ee707a9c89986cdeea2aeef"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.623456 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:32.624244 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.624434 10449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:32.624533 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "bf00cbc4ecad4a59a01874f0734e6567"
format_stamp: "Formatted at 2026-09-28 17:52:32 on dist-test-slave-zqlt"
server_key: "8172f6677ed014be479d80188fca511f"
server_key_iv: "15d388b93ee707a9c89986cdeea2aeef"
server_key_version: "encryptionkey@0"
I20260928 17:52:32.624619 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:32.628037 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:32.628386 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:32.628544 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:32.628767 10449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:32.629106 10449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:32.629160 10449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.629201 10449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:32.629227 10449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:32.635388 10449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42889
I20260928 17:52:32.635423 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42889 every 8 connection(s)
I20260928 17:52:32.635800 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:52:32.638989 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.636603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33399 (local address 127.0.98.62:40071)
0928 17:52:32.636660 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.636663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.636906 (+ 243us) server_negotiation.cc:407] Connection header received
0928 17:52:32.637399 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.637401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.637407 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.637478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.638692 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:32.638694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:32.638694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:52:32.638698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:32.638714 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:52:32.638718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:32.638776 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:32.638916 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:52:32.638931 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:52:32.639262 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:32.636556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40071 (local address 127.0.98.3:33399)
0928 17:52:32.636746 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:32.636790 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:32.637293 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:32.637553 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:32.637557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:32.637566 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:32.638532 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:52:32.638532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:32.638563 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:52:32.638593 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:32.638814 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:32.638820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:32.638824 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:52:32.638913 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:52:32.638975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260928 17:52:32.639267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10449
I20260928 17:52:32.639356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:32.639588 392 external_mini_cluster.cc:1428] Setting key ab58dc4d54fa3e946db7aa32a5e07b35
I20260928 17:52:32.639770 10577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40071
I20260928 17:52:32.639896 10577 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:32.640064 10577 heartbeater.cc:511] Master 127.0.98.62:40071 requested a full tablet report, sending...
I20260928 17:52:32.640497 10126 ts_manager.cc:194] Registered new tserver with Master: bf00cbc4ecad4a59a01874f0734e6567 (127.0.98.3:42889)
I20260928 17:52:32.651700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 17:52:32.653728 10184 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:52:32.653252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.0.98.62:40071)
0928 17:52:32.653330 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:32.653332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:32.653384 (+ 52us) server_negotiation.cc:407] Connection header received
0928 17:52:32.653436 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:32.653437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:32.653441 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:32.653490 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:32.653646 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56976: BlockingRecv error: recv got EOF from 127.0.0.1:56976 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 17:52:32.653769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10187
I20260928 17:52:32.658442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10318
I20260928 17:52:32.662931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10449
I20260928 17:52:32.667253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10096
2026-09-28T17:52:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (574 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:32 dist-test-slave-zqlt krb5kdc[10587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:32 dist-test-slave-zqlt krb5kdc[10587](info): set up 2 sockets
Sep 28 17:52:32 dist-test-slave-zqlt krb5kdc[10587](info): commencing operation
krb5kdc: starting...
W20260928 17:52:33.709189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:33 dist-test-slave-zqlt krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:52:33.823119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:33.905820 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:33.906035 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:33.906059 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:33.907532 10600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:52:33.907586 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:33.907601 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:33.907646 10600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:52:33.907673 10600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:52:33.909302 10600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:33.909559 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:33.909789 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:33.914937 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:33.914943 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:33.915006 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:33.915222 10600 server_base.cc:1034] running on GCE node
I20260928 17:52:33.915499 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:33.915831 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:33.916996 10600 hybrid_clock.cc:648] HybridClock initialized: now 1790617953916965 us; error 47 us; skew 500 ppm
Sep 28 17:52:33 dist-test-slave-zqlt krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:33.919529 10600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:52:33.919960 10600 webserver.cc:460] Webserver started at http://127.0.98.62:44401/ using document root <none> and password file <none>
I20260928 17:52:33.920256 10600 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:33.920334 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:33.920492 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:33.921540 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "f5e38ffdc2664edab6268ef852102968"
format_stamp: "Formatted at 2026-09-28 17:52:33 on dist-test-slave-zqlt"
server_key: "954fddf22813a4f976146e206bb0be74"
server_key_iv: "83843086a2aa9374f76bbb20a4ba6b8a"
server_key_version: "encryptionkey@0"
I20260928 17:52:33.921902 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5e38ffdc2664edab6268ef852102968"
format_stamp: "Formatted at 2026-09-28 17:52:33 on dist-test-slave-zqlt"
server_key: "954fddf22813a4f976146e206bb0be74"
server_key_iv: "83843086a2aa9374f76bbb20a4ba6b8a"
server_key_version: "encryptionkey@0"
I20260928 17:52:33.923239 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:52:33.923940 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:33.924135 10600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:33.924247 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "f5e38ffdc2664edab6268ef852102968"
format_stamp: "Formatted at 2026-09-28 17:52:33 on dist-test-slave-zqlt"
server_key: "954fddf22813a4f976146e206bb0be74"
server_key_iv: "83843086a2aa9374f76bbb20a4ba6b8a"
server_key_version: "encryptionkey@0"
I20260928 17:52:33.924363 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:33.940788 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:33.943181 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:33.943360 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:33.947494 10600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41551
I20260928 17:52:33.947535 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41551 every 8 connection(s)
I20260928 17:52:33.947888 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:52:33.948778 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:33.951143 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: Bootstrap starting.
I20260928 17:52:33.951759 10668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:33.952065 10668 log.cc:824] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:33.952797 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: No bootstrap required, opened a new log
I20260928 17:52:33.954384 10668 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260928 17:52:33.954528 10668 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:33.954550 10668 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38ffdc2664edab6268ef852102968, State: Initialized, Role: FOLLOWER
I20260928 17:52:33.954737 10668 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260928 17:52:33.954803 10668 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:52:33.954829 10668 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:52:33.954907 10668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:33.955724 10668 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260928 17:52:33.955811 10668 leader_election.cc:302] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e38ffdc2664edab6268ef852102968; no voters:
I20260928 17:52:33.955974 10668 leader_election.cc:288] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:52:33.956022 10671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:33.956238 10671 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [term 1 LEADER]: Becoming Leader. State: Replica: f5e38ffdc2664edab6268ef852102968, State: Running, Role: LEADER
I20260928 17:52:33.956329 10668 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:52:33.956382 10671 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } }
I20260928 17:52:33.956786 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e38ffdc2664edab6268ef852102968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260928 17:52:33.956871 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:33.956825 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e38ffdc2664edab6268ef852102968. Latest consensus state: current_term: 1 leader_uuid: "f5e38ffdc2664edab6268ef852102968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e38ffdc2664edab6268ef852102968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41551 } } }
I20260928 17:52:33.957038 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968 [sys.catalog]: This master's current role is: LEADER
I20260928 17:52:33.957124 10679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:52:33.957651 10679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:52:33.957890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10600
I20260928 17:52:33.958006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:52:33.958204 392 external_mini_cluster.cc:1428] Setting key bf65f7d802398ed35c3e440a419a945e
I20260928 17:52:33.959877 10679 catalog_manager.cc:1347] Generated new cluster ID: 1036ffce158743fbac2c33f712c9d3c5
I20260928 17:52:33.959935 10679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:52:33 dist-test-slave-zqlt krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:33.967703 10679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:52:33.968431 10679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:52:33.968653 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:33.958720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.0.98.62:41551)
0928 17:52:33.958883 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:33.958893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:33.959088 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:52:33.959222 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:33.959230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:33.959288 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:33.959438 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:33.960109 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:33.961064 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:33.961834 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:33.962285 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:33.962411 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:33.964800 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:33.964804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:33.964804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:33.964817 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:33.966550 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:33.966940 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:33.966945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:33.966946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:33.967046 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:33.967468 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:33.967469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:33.967470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:33.967783 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:33.967945 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:33.968182 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:52:33.968394 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 17:52:33.980027 10679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: Generated new TSK 0
I20260928 17:52:33.980298 10679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:52:34.011140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:34.112772 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:34.113003 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:34.113044 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:34.114219 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:34.114284 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:34.114334 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:52:34.115837 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:34.116153 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:34.116401 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:34.121799 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:34.121846 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:34.121799 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:34.122021 10694 server_base.cc:1034] running on GCE node
I20260928 17:52:34.122283 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:34.122547 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:34.123698 10694 hybrid_clock.cc:648] HybridClock initialized: now 1790617954123672 us; error 41 us; skew 500 ppm
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:34.126526 10694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:52:34.127059 10694 webserver.cc:460] Webserver started at http://127.0.98.1:37115/ using document root <none> and password file <none>
I20260928 17:52:34.127254 10694 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:34.127295 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:34.127398 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:34.128589 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb0dea5ff116435c84e11ff359009d21"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "f6513412d499d4b0b4e613ac5f32c98d"
server_key_iv: "84400e06fca8da48f5fc2ab151fc7e3b"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.128965 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb0dea5ff116435c84e11ff359009d21"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "f6513412d499d4b0b4e613ac5f32c98d"
server_key_iv: "84400e06fca8da48f5fc2ab151fc7e3b"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.130342 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:34.131081 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.131246 10694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:34.131366 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "bb0dea5ff116435c84e11ff359009d21"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "f6513412d499d4b0b4e613ac5f32c98d"
server_key_iv: "84400e06fca8da48f5fc2ab151fc7e3b"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.131440 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:34.147769 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:34.150370 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:34.150547 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:34.150791 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:34.151129 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:34.151185 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.151229 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:34.151253 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.157631 10694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34701
I20260928 17:52:34.157688 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34701 every 8 connection(s)
I20260928 17:52:34.158092 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:34.166906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10694
I20260928 17:52:34.166996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:52:34.167212 392 external_mini_cluster.cc:1428] Setting key dc7b1e38feb3fe9a9ecc39867518e3a7
I20260928 17:52:34.167701 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.158923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59165 (local address 127.0.98.62:41551)
0928 17:52:34.159016 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.159019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.159154 (+ 135us) server_negotiation.cc:407] Connection header received
0928 17:52:34.159713 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.159714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.159726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.159789 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:34.160838 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.161348 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.162320 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.162522 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.162579 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.164827 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:34.164829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:34.164830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:34.164838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:34.166097 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.166504 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.166506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.166506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.166561 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.166913 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.166914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.166914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.167081 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:34.167140 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.167497 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.167610 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 17:52:34.167863 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.158858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.1:59165)
0928 17:52:34.159036 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.159075 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.159577 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.159849 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.159853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.160528 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:34.160733 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.160738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.161436 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.161437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.162241 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.162243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.162598 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.162598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.162699 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.163169 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:34.163177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:34.164728 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:34.166185 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.166192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.166194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.166426 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.166638 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.166639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.166640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.166808 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.167139 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:34.167146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:34.167316 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.167457 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.167649 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 17:52:34.168334 10823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260928 17:52:34.168442 10823 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:34.168642 10823 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260928 17:52:34.169215 10632 ts_manager.cc:194] Registered new tserver with Master: bb0dea5ff116435c84e11ff359009d21 (127.0.98.1:34701)
I20260928 17:52:34.169864 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59165
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:52:34.190802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:34.270149 10828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:34.270357 10828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:34.270386 10828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:34.271559 10828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:34.271601 10828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:34.271656 10828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:52:34.273149 10828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:34.273401 10828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:34.273651 10828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:34.278861 10833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:34.278879 10828 server_base.cc:1034] running on GCE node
W20260928 17:52:34.278880 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:34.278861 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:34.279313 10828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:34.279573 10828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:34.280712 10828 hybrid_clock.cc:648] HybridClock initialized: now 1790617954280693 us; error 40 us; skew 500 ppm
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:34.283216 10828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:52:34.283643 10828 webserver.cc:460] Webserver started at http://127.0.98.2:38777/ using document root <none> and password file <none>
I20260928 17:52:34.283831 10828 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:34.283872 10828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:34.283967 10828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:34.285099 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a34317310b04f03b761301270cf4a10"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "43fc986cfa6097f629a409c406ff3db0"
server_key_iv: "03bb337ea9174d8929ceb7b2426ee2ba"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.285408 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a34317310b04f03b761301270cf4a10"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "43fc986cfa6097f629a409c406ff3db0"
server_key_iv: "03bb337ea9174d8929ceb7b2426ee2ba"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.286706 10828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:34.287472 10843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.287758 10828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:52:34.287865 10828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "2a34317310b04f03b761301270cf4a10"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "43fc986cfa6097f629a409c406ff3db0"
server_key_iv: "03bb337ea9174d8929ceb7b2426ee2ba"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.287945 10828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:34.296046 10828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:34.298012 10828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:34.298125 10828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:34.298316 10828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:34.298578 10828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:34.298609 10828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.298631 10828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:34.298645 10828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.304524 10828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41309
I20260928 17:52:34.304598 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41309 every 8 connection(s)
I20260928 17:52:34.304942 10828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:52:34.305101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10828
I20260928 17:52:34.305207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:52:34.305441 392 external_mini_cluster.cc:1428] Setting key 69d6b246d04abddc038e23ee2cd5179a
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:52:34.316313 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.305745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49897 (local address 127.0.98.62:41551)
0928 17:52:34.305793 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.305795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.306131 (+ 336us) server_negotiation.cc:407] Connection header received
0928 17:52:34.306904 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.306905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.306915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.306964 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:34.308351 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.308870 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.309987 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.310204 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.310294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.312838 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:34.312840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:34.312840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:34.312849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:34.314837 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.315267 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.315268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.315269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.315329 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.315567 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.315568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.315568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.315751 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:34.315834 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.316048 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.316183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 17:52:34.316592 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.305738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.2:49897)
0928 17:52:34.305981 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.306026 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.306784 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.307041 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.307046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.307967 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:34.308257 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.308262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.308985 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.308986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.309902 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.309905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.310303 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.310304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.310439 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.311040 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:34.311048 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:34.312709 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:34.314945 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.314950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.314952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.315173 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.315409 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.315410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.315410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.315507 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.315801 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:34.315805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:34.315931 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.316018 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.316272 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260928 17:52:34.317091 10957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260928 17:52:34.317188 10957 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:34.317394 10957 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260928 17:52:34.317956 10632 ts_manager.cc:194] Registered new tserver with Master: 2a34317310b04f03b761301270cf4a10 (127.0.98.2:41309)
I20260928 17:52:34.318387 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:52:34.330874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:52:34.414021 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:52:34.414232 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:52:34.414254 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:52:34.415352 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:52:34.415388 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:52:34.415414 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:52:34.416800 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:52:34.417085 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:52:34.417333 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:52:34.422467 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:34.422508 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:52:34.422467 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:52:34.422796 10962 server_base.cc:1034] running on GCE node
I20260928 17:52:34.422963 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:52:34.423190 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:52:34.424326 10962 hybrid_clock.cc:648] HybridClock initialized: now 1790617954424314 us; error 27 us; skew 500 ppm
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:52:34.427093 10962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:52:34.427506 10962 webserver.cc:460] Webserver started at http://127.0.98.3:39209/ using document root <none> and password file <none>
I20260928 17:52:34.427692 10962 fs_manager.cc:359] Metadata directory not provided
I20260928 17:52:34.427731 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:52:34.427835 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:52:34.428932 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "37283db5589a4388a1f294f4de4a4cd4"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "6790d09d8797929875bf8e47f2b3f92f"
server_key_iv: "dc2c7b19e3ad7a2fbdd756a589765322"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.429237 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37283db5589a4388a1f294f4de4a4cd4"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "6790d09d8797929875bf8e47f2b3f92f"
server_key_iv: "dc2c7b19e3ad7a2fbdd756a589765322"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.430521 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:52:34.431193 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.431365 10962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:52:34.431438 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "37283db5589a4388a1f294f4de4a4cd4"
format_stamp: "Formatted at 2026-09-28 17:52:34 on dist-test-slave-zqlt"
server_key: "6790d09d8797929875bf8e47f2b3f92f"
server_key_iv: "dc2c7b19e3ad7a2fbdd756a589765322"
server_key_version: "encryptionkey@0"
I20260928 17:52:34.431496 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:52:34.448628 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:52:34.450515 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:52:34.450629 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:52:34.450815 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:52:34.451133 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:52:34.451166 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.451220 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:52:34.451246 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:52:34.457248 10962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44735
I20260928 17:52:34.457316 11090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44735 every 8 connection(s)
I20260928 17:52:34.457638 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:52:34.466078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 10962
I20260928 17:52:34.466202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:52:34.466423 392 external_mini_cluster.cc:1428] Setting key 4dbafab7adbdb8b25f95a46dd899d305
I20260928 17:52:34.467283 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.458508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56395 (local address 127.0.98.62:41551)
0928 17:52:34.458588 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.458591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.458831 (+ 240us) server_negotiation.cc:407] Connection header received
0928 17:52:34.459331 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.459332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.459342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.459391 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:34.460422 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.460912 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.461883 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.462061 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.462114 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.464315 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:34.464317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:34.464317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:34.464324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:34.465735 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.466123 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.466125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.466126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.466209 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.466476 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.466477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.466478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.466639 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:34.466710 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.467121 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.467206 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:52:34.467523 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.458486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41551 (local address 127.0.98.3:56395)
0928 17:52:34.458660 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.458700 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.459240 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.459457 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.459460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.460066 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:52:34.460334 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.460339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.461002 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.461003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.461802 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.461804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.462135 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.462135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.462235 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.462766 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:52:34.462774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:52:34.464246 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:52:34.465805 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.465814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.465816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.466039 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.466279 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:52:34.466280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:52:34.466281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:52:34.466397 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:52:34.466775 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:52:34.466781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:52:34.466944 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.467091 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.467264 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260928 17:52:34.467959 11091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41551
I20260928 17:52:34.468081 11091 heartbeater.cc:464] Registering TS with master...
I20260928 17:52:34.468288 11091 heartbeater.cc:511] Master 127.0.98.62:41551 requested a full tablet report, sending...
I20260928 17:52:34.468714 10632 ts_manager.cc:194] Registered new tserver with Master: 37283db5589a4388a1f294f4de4a4cd4 (127.0.98.3:44735)
I20260928 17:52:34.469101 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56395
I20260928 17:52:34.478224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:52:34.485473 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.479614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.98.62:41551)
0928 17:52:34.479668 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.479671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.479813 (+ 142us) server_negotiation.cc:407] Connection header received
0928 17:52:34.479882 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.479883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.479896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.479949 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:52:34.480611 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.481114 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.481810 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.482000 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.482049 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.483159 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:52:34.483160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:52:34.483160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:52:34.483167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:52:34.484477 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.484804 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.484805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.484805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.484854 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:52:34.485037 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:52:34.485037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:52:34.485037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:52:34.485171 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:52:34.485246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.485352 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.485409 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 17:52:34.487054 10632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:52:34.488668 10632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:52:34.494894 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.490892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44735 (local address 127.0.0.1:54608)
0928 17:52:34.491228 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.491238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.491276 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.491528 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.491529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.491531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:34.491579 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.491581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492874 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.492875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.493875 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.493877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.494581 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.494582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.494660 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.494674 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.494751 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.494802 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":45,"thread_start_us":68,"threads_started":1}
I20260928 17:52:34.494907 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.490968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.3:44735)
0928 17:52:34.491269 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.491273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.491282 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:52:34.491346 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.491348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.491364 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.491440 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:34.491658 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492773 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.493961 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.494499 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.494575 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.494633 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.494776 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.494819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":83,"threads_started":1}
I20260928 17:52:34.495391 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.491066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34701 (local address 127.0.0.1:40376)
0928 17:52:34.491451 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.491461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.491503 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.491916 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.491917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.491919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:34.491975 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.491977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492882 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.492883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.494144 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.494146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.495114 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.495115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.495180 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.495183 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.495261 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.495310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":83,"thread_start_us":66,"threads_started":1}
I20260928 17:52:34.495422 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.491283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.98.1:34701)
0928 17:52:34.491637 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.491642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.491652 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:34.491808 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.491810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.491828 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.491917 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:34.492031 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492750 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.494230 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.494949 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.495020 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.495083 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.495282 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.495328 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":81,"threads_started":1}
I20260928 17:52:34.495816 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.491060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.0.98.2:41309)
0928 17:52:34.491212 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.491226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.491238 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:52:34.491282 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.491284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.491302 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.491385 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:34.491543 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492724 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.494447 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.495176 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.495270 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.495324 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.495683 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.495735 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 17:52:34.495847 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.490878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41309 (local address 127.0.0.1:52360)
0928 17:52:34.491025 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.491064 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.491126 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.491402 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.491405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.491407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:34.491475 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.491477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.492833 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.492834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.494353 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.494355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.495511 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.495513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.495583 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.495586 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.495667 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.495710 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":74,"thread_start_us":46,"threads_started":1}
I20260928 17:52:34.496176 10757 tablet_service.cc:1467] Processing CreateTablet for tablet 98a450ced9bd48139739fcfcc111babd (DEFAULT_TABLE table=test-table [id=1841e612b2c048499dc313510b3ad696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:34.496178 11025 tablet_service.cc:1467] Processing CreateTablet for tablet 98a450ced9bd48139739fcfcc111babd (DEFAULT_TABLE table=test-table [id=1841e612b2c048499dc313510b3ad696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:34.496544 10757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a450ced9bd48139739fcfcc111babd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:34.496544 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a450ced9bd48139739fcfcc111babd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:34.499943 11108 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Bootstrap starting.
I20260928 17:52:34.500128 11109 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Bootstrap starting.
I20260928 17:52:34.500723 11109 tablet_bootstrap.cc:654] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:34.500694 11108 tablet_bootstrap.cc:654] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:34.500676 10891 tablet_service.cc:1467] Processing CreateTablet for tablet 98a450ced9bd48139739fcfcc111babd (DEFAULT_TABLE table=test-table [id=1841e612b2c048499dc313510b3ad696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:52:34.501014 10891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a450ced9bd48139739fcfcc111babd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:52:34.501312 11109 log.cc:824] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:34.501312 11108 log.cc:824] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:34.502064 11109 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: No bootstrap required, opened a new log
I20260928 17:52:34.502120 11109 ts_tablet_manager.cc:1397] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:52:34.503309 11108 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: No bootstrap required, opened a new log
I20260928 17:52:34.503379 11108 ts_tablet_manager.cc:1397] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 17:52:34.503753 11109 raft_consensus.cc:348] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.503933 11109 raft_consensus.cc:374] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:34.503970 11109 raft_consensus.cc:729] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37283db5589a4388a1f294f4de4a4cd4, State: Initialized, Role: FOLLOWER
I20260928 17:52:34.503887 11112 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Bootstrap starting.
I20260928 17:52:34.504079 11109 consensus_queue.cc:260] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.504362 11109 ts_tablet_manager.cc:1428] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 17:52:34.504483 11091 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
I20260928 17:52:34.504743 11112 tablet_bootstrap.cc:654] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Neither blocks nor log segments found. Creating new log.
I20260928 17:52:34.505278 11112 log.cc:824] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Log is configured to *not* fsync() on all Append() calls
I20260928 17:52:34.505236 11108 raft_consensus.cc:348] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.505414 11108 raft_consensus.cc:374] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:34.505468 11108 raft_consensus.cc:729] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0dea5ff116435c84e11ff359009d21, State: Initialized, Role: FOLLOWER
I20260928 17:52:34.505586 11108 consensus_queue.cc:260] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.505856 11108 ts_tablet_manager.cc:1428] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:52:34.505883 10823 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
I20260928 17:52:34.506027 11112 tablet_bootstrap.cc:492] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: No bootstrap required, opened a new log
I20260928 17:52:34.506076 11112 ts_tablet_manager.cc:1397] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:52:34.507607 11112 raft_consensus.cc:348] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.507735 11112 raft_consensus.cc:374] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:52:34.507758 11112 raft_consensus.cc:729] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a34317310b04f03b761301270cf4a10, State: Initialized, Role: FOLLOWER
I20260928 17:52:34.507863 11112 consensus_queue.cc:260] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.508139 11112 ts_tablet_manager.cc:1428] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:52:34.508179 10957 heartbeater.cc:503] Master 127.0.98.62:41551 was elected leader, sending a full tablet report...
W20260928 17:52:34.555641 10958 tablet.cc:2369] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:52:34.658902 10824 tablet.cc:2369] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:34.687618 11116 raft_consensus.cc:482] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:52:34.687798 11116 raft_consensus.cc:504] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.688226 11116 leader_election.cc:288] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37283db5589a4388a1f294f4de4a4cd4 (127.0.98.3:44735), bb0dea5ff116435c84e11ff359009d21 (127.0.98.1:34701)
I20260928 17:52:34.691217 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.688473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44735 (local address 127.0.98.2:37803)
0928 17:52:34.688546 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.688562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.688633 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.688769 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.688770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.688771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:34.688843 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.688845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.689513 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.689513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.690380 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.690383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.690943 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.690944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.691030 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.691033 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.691096 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.691137 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260928 17:52:34.691229 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.688582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37803 (local address 127.0.98.3:44735)
0928 17:52:34.688650 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.688652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.688662 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:52:34.688692 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.688693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.688705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.688761 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:34.688937 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.689416 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.690473 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.690849 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.690944 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.690954 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.691114 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.691155 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 17:52:34.691500 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.688677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49101 (local address 127.0.98.1:34701)
0928 17:52:34.688731 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.688733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.688872 (+ 139us) server_negotiation.cc:407] Connection header received
0928 17:52:34.688954 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.688955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.688972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.689045 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:52:34.689195 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.689665 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.690646 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.691085 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.691167 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.691176 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.691377 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.691433 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 17:52:34.691609 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a450ced9bd48139739fcfcc111babd" candidate_uuid: "2a34317310b04f03b761301270cf4a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37283db5589a4388a1f294f4de4a4cd4" is_pre_election: true
I20260928 17:52:34.691730 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.688616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34701 (local address 127.0.98.2:49101)
0928 17:52:34.688784 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.688796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:52:34.688886 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:52:34.689046 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:52:34.689047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:52:34.689049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:52:34.689124 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.689125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.689757 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.689758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.690551 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:52:34.690553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.691167 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.691168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:52:34.691259 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.691262 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:52:34.691622 (+ 360us) client_negotiation.cc:240] Negotiation successful
0928 17:52:34.691665 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":156,"threads_started":1}
I20260928 17:52:34.691798 11045 raft_consensus.cc:2454] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a34317310b04f03b761301270cf4a10 in term 0.
I20260928 17:52:34.692102 10847 leader_election.cc:302] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a34317310b04f03b761301270cf4a10, 37283db5589a4388a1f294f4de4a4cd4; no voters:
I20260928 17:52:34.692183 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a450ced9bd48139739fcfcc111babd" candidate_uuid: "2a34317310b04f03b761301270cf4a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0dea5ff116435c84e11ff359009d21" is_pre_election: true
I20260928 17:52:34.692272 11116 raft_consensus.cc:2781] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:52:34.692324 11116 raft_consensus.cc:482] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:52:34.692373 11116 raft_consensus.cc:3037] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:34.692374 10777 raft_consensus.cc:2454] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a34317310b04f03b761301270cf4a10 in term 0.
I20260928 17:52:34.693202 11116 raft_consensus.cc:504] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.693338 11116 leader_election.cc:288] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [CANDIDATE]: Term 1 election: Requested vote from peers 37283db5589a4388a1f294f4de4a4cd4 (127.0.98.3:44735), bb0dea5ff116435c84e11ff359009d21 (127.0.98.1:34701)
I20260928 17:52:34.693538 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a450ced9bd48139739fcfcc111babd" candidate_uuid: "2a34317310b04f03b761301270cf4a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37283db5589a4388a1f294f4de4a4cd4"
I20260928 17:52:34.693594 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a450ced9bd48139739fcfcc111babd" candidate_uuid: "2a34317310b04f03b761301270cf4a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0dea5ff116435c84e11ff359009d21"
I20260928 17:52:34.693643 11045 raft_consensus.cc:3037] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:34.693693 10777 raft_consensus.cc:3037] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:52:34.694371 10777 raft_consensus.cc:2454] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a34317310b04f03b761301270cf4a10 in term 1.
I20260928 17:52:34.694381 11045 raft_consensus.cc:2454] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a34317310b04f03b761301270cf4a10 in term 1.
I20260928 17:52:34.694553 10847 leader_election.cc:302] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a34317310b04f03b761301270cf4a10, 37283db5589a4388a1f294f4de4a4cd4; no voters:
I20260928 17:52:34.694684 11116 raft_consensus.cc:2781] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:52:34.694867 11116 raft_consensus.cc:686] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 1 LEADER]: Becoming Leader. State: Replica: 2a34317310b04f03b761301270cf4a10, State: Running, Role: LEADER
I20260928 17:52:34.694981 11116 consensus_queue.cc:237] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } }
I20260928 17:52:34.695883 10632 catalog_manager.cc:5495] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a34317310b04f03b761301270cf4a10 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2a34317310b04f03b761301270cf4a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a34317310b04f03b761301270cf4a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 } health_report { overall_health: UNKNOWN } } }
W20260928 17:52:34.708314 11092 tablet.cc:2369] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:52:34.725128 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.722615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.98.2:41309)
0928 17:52:34.722674 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:52:34.722676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:52:34.722688 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:52:34.722751 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:52:34.722752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:52:34.722772 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:52:34.722865 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:52:34.722992 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.723474 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.724294 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:52:34.724503 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:52:34.724563 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:52:34.724704 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:52:34.724813 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:52:34.724885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:52:34.724969 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:52:34.724989 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:52:34.727375 11045 raft_consensus.cc:1261] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 1 FOLLOWER]: Refusing update from remote peer 2a34317310b04f03b761301270cf4a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:34.727396 10777 raft_consensus.cc:1261] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 1 FOLLOWER]: Refusing update from remote peer 2a34317310b04f03b761301270cf4a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:52:34.727836 11118 consensus_queue.cc:1035] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37283db5589a4388a1f294f4de4a4cd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:34.727986 11116 consensus_queue.cc:1035] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0dea5ff116435c84e11ff359009d21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:52:34.730448 11121 mvcc.cc:205] Tried to move back new op lower bound from 7334371142559449088 to 7334371142431277056. Current Snapshot: MvccSnapshot[applied={T|T < 7334371142559449088}]
I20260928 17:52:34.733559 10632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:34.733670 10632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:52:34.734571 10632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5e38ffdc2664edab6268ef852102968: Sending DeleteTablet for 3 replicas of tablet 98a450ced9bd48139739fcfcc111babd
I20260928 17:52:34.734887 10891 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a450ced9bd48139739fcfcc111babd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52360
I20260928 17:52:34.734953 11025 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a450ced9bd48139739fcfcc111babd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54608
I20260928 17:52:34.735028 10757 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a450ced9bd48139739fcfcc111babd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:52:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40376
I20260928 17:52:34.735244 11132 tablet_replica.cc:330] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21: stopping tablet replica
I20260928 17:52:34.735301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10694
I20260928 17:52:34.735347 11132 raft_consensus.cc:2229] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:34.735502 11132 raft_consensus.cc:2258] T 98a450ced9bd48139739fcfcc111babd P bb0dea5ff116435c84e11ff359009d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:34.735642 11131 tablet_replica.cc:330] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: stopping tablet replica
I20260928 17:52:34.735725 11131 raft_consensus.cc:2229] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:52:34.735801 11131 raft_consensus.cc:2258] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:34.736042 11130 tablet_replica.cc:330] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: stopping tablet replica
I20260928 17:52:34.736164 11130 raft_consensus.cc:2229] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:52:34.736294 11130 raft_consensus.cc:2258] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:52:34.736573 11131 ts_tablet_manager.cc:1905] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:34.736658 11130 ts_tablet_manager.cc:1905] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:52:34.738366 11130 ts_tablet_manager.cc:1918] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:34.738446 11130 log.cc:1196] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/98a450ced9bd48139739fcfcc111babd
I20260928 17:52:34.738723 11130 ts_tablet_manager.cc:1939] T 98a450ced9bd48139739fcfcc111babd P 2a34317310b04f03b761301270cf4a10: Deleting consensus metadata
I20260928 17:52:34.738862 11131 ts_tablet_manager.cc:1918] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:52:34.738979 11131 log.cc:1196] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/98a450ced9bd48139739fcfcc111babd
I20260928 17:52:34.739176 10616 catalog_manager.cc:4841] TS 2a34317310b04f03b761301270cf4a10 (127.0.98.2:41309): tablet 98a450ced9bd48139739fcfcc111babd (table test-table [id=1841e612b2c048499dc313510b3ad696]) successfully deleted
I20260928 17:52:34.739392 11131 ts_tablet_manager.cc:1939] T 98a450ced9bd48139739fcfcc111babd P 37283db5589a4388a1f294f4de4a4cd4: Deleting consensus metadata
I20260928 17:52:34.739842 10619 catalog_manager.cc:4841] TS 37283db5589a4388a1f294f4de4a4cd4 (127.0.98.3:44735): tablet 98a450ced9bd48139739fcfcc111babd (table test-table [id=1841e612b2c048499dc313510b3ad696]) successfully deleted
W20260928 17:52:34.742815 10619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34701 (error 108)
I20260928 17:52:34.743378 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:52:34.743236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34701 (local address 127.0.0.1:40380)
0928 17:52:34.743298 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 17:52:34.743342 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:52:34.743399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10828
W20260928 17:52:34.744781 10619 catalog_manager.cc:4584] TS bb0dea5ff116435c84e11ff359009d21 (127.0.98.1:34701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98a450ced9bd48139739fcfcc111babd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34701: connect: Connection refused (error 111)
I20260928 17:52:34.748689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10962
I20260928 17:52:34.753520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 10600
2026-09-28T17:52:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2086 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[11136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[11136](info): set up 2 sockets
Sep 28 17:52:34 dist-test-slave-zqlt krb5kdc[11136](info): commencing operation
krb5kdc: starting...
W20260928 17:52:35.825398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:52:35 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:52:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 17:52:35.982215 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMA1V5K/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/postgres -l logfile start
2026-09-28 17:52:38.431 UTC [11166] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 17:52:38.431 UTC [11166] LOG: listening on IPv4 address "127.0.98.20", port 39991
2026-09-28 17:52:38.435 UTC [11166] LOG: listening on Unix socket "/tmp/.s.PGSQL.39991"
2026-09-28 17:52:38.470 UTC [11169] LOG: database system was shut down at 2026-09-28 17:52:37 UTC
2026-09-28 17:52:38.475 UTC [11166] LOG: database system is ready to accept connections
I20260928 17:52:39.421047 392 mini_postgres.cc:96] Postgres bound to 39991
2026-09-28 17:52:39.427 UTC [11177] FATAL: database "slave" does not exist
127.0.98.20:39991 - accepting connections
I20260928 17:52:39.427814 392 mini_ranger.cc:161] Starting Ranger...
I20260928 17:52:39.441958 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 17:52:39.941761 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 17:52:39.941851 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin
2026-09-28 17:52:40,214 [I] DB FLAVOR :POSTGRES
2026-09-28 17:52:40,215 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:52:40,215 [I] Checking connection..
2026-09-28 17:52:40,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 17:52:40,455 [I] Checking connection passed.
2026-09-28 17:52:40,455 [I] --------- Verifying version history table ---------
2026-09-28 17:52:40,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:52:40,704 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:52:40,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:52:40,951 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:52:40,951 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 17:52:40,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 17:52:41.210 UTC [11241] WARNING: there is no transaction in progress
2026-09-28 17:52:41,216 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 17:52:41,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:52:41,450 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 17:52:41,450 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:52:41,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 17:52:41,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 17:52:41,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zqlt.c.gcp-upstream.internal','N') ;"
2026-09-28 17:52:42,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:52:42,399 [I] Table x_portal_user does not exist in database ranger
2026-09-28 17:52:42,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:52:42,643 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 17:52:42,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:52:42,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:52:43,128 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 17:52:43,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 17:52:44.052 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.068 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.085 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.100 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.115 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.133 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.188 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.194 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.208 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.218 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.228 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.243 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.252 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.259 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.266 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44.273 UTC [11349] WARNING: there is no transaction in progress
2026-09-28 17:52:44,925 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 17:52:44,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 17:52:45,175 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 17:52:45,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:52:45,416 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 17:52:45,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:52:45,697 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 17:52:45,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:52:45,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:52:46,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zqlt.c.gcp-upstream.internal';"
2026-09-28 17:52:46,420 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 17:52:46,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 17:52:46,663 [I] DB_PATCHES have already been applied
I20260928 17:52:46.669063 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMA1V5K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 17:52:46.669126 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 17:52:46.672135 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44567
Sep 28, 2026 5:52:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:52:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:52:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:52:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:52:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:52:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:52:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 17:52:47.791023 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.363s user 0.003s sys 0.006s
Sep 28, 2026 5:53:08 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:53:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:53:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:53:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:53:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 17:53:10.003310 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 17:53:10 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 17:53:10.090788 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 17:53:10.100418 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 17:53:10.952463 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 17:53:10.952549 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms
I20260928 17:53:10.952571 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39991
2026-09-28 17:53:11,128 [I] DB FLAVOR :POSTGRES
2026-09-28 17:53:11,129 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:53:11,129 [I] Checking connection
2026-09-28 17:53:11,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 17:53:11,391 [I] connection success
2026-09-28 17:53:11,391 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 17:53:11,391 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 17:53:11,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 17:53:11,709 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 17:53:11,710 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:53:11,710 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 17:53:11,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39991/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 17:53:12,000 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 17:53:12.482369 392 mini_ranger_kms.cc:310] Created kms service
I20260928 17:53:12.717280 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 17:53:12.763537 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 17:53:12 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:53:13.155191 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 17:53:13.155306 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 17:53:13.155330 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 17:53:13.158694 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11529
I20260928 17:53:13.158835 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56481
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:53:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 17:53:15.388314 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.298s user 0.000s sys 0.159s
I20260928 17:53:15.388410 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 17:53:15.388484 392 mini_ranger_kms.cc:401] 127.0.98.20:56481/kms/v1/keys
Sep 28 17:53:15 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790617995, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 17:53:15 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617995, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 17:53:16 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617995, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 17:53:18 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:53:18.807457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40235
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56481/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:53:18.908299 11580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:53:18.908497 11580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:53:18.908519 11580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:53:18.909667 11580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:53:18.909710 11580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:53:18.909725 11580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:53:18.909737 11580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:53:18.909791 11580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:53:18.911314 11580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40235
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:53:18.911554 11580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:53:18.911819 11580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:53:18.918059 11586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:18.918056 11588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:53:18.918171 11580 server_base.cc:1034] running on GCE node
W20260928 17:53:18.918332 11585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:53:18.918596 11580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:53:18.918972 11580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:53:18.920153 11580 hybrid_clock.cc:648] HybridClock initialized: now 1790617998920136 us; error 34 us; skew 500 ppm
Sep 28 17:53:18 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:53:18.922904 11580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:53:18.923457 11580 webserver.cc:460] Webserver started at http://127.0.98.62:44711/ using document root <none> and password file <none>
I20260928 17:53:18.923707 11580 fs_manager.cc:359] Metadata directory not provided
I20260928 17:53:18.923767 11580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:53:18.923933 11580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:53:18 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:53:18.977316 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "14f4b3908ef04ab796d86e6a4f1d4156"
format_stamp: "Formatted at 2026-09-28 17:53:18 on dist-test-slave-zqlt"
server_key: "6f3e3f4c5b47763ec3ebd08e232f03bc"
server_key_iv: "c300ee10a32d8894026f89ed6c803841"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:18.977736 11580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "14f4b3908ef04ab796d86e6a4f1d4156"
format_stamp: "Formatted at 2026-09-28 17:53:18 on dist-test-slave-zqlt"
server_key: "6f3e3f4c5b47763ec3ebd08e232f03bc"
server_key_iv: "c300ee10a32d8894026f89ed6c803841"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:18.979178 11580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:53:18.993053 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:18.993287 11580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:53:18.993417 11580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "14f4b3908ef04ab796d86e6a4f1d4156"
format_stamp: "Formatted at 2026-09-28 17:53:18 on dist-test-slave-zqlt"
server_key: "6f3e3f4c5b47763ec3ebd08e232f03bc"
server_key_iv: "c300ee10a32d8894026f89ed6c803841"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:18.993541 11580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:53:19.002182 11580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:53:19.004400 11580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:53:19.004572 11580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:53:19.009219 11580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40235
I20260928 17:53:19.009268 11647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40235 every 8 connection(s)
I20260928 17:53:19.009656 11580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:53:19.010403 11648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:53:19.013396 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: Bootstrap starting.
I20260928 17:53:19.014081 11648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: Neither blocks nor log segments found. Creating new log.
I20260928 17:53:19.014282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 11580
I20260928 17:53:19.014358 11648 log.cc:824] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: Log is configured to *not* fsync() on all Append() calls
I20260928 17:53:19.014400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:53:19.014618 392 external_mini_cluster.cc:1428] Setting key 45141566716d5c14e9c1faa409052996
I20260928 17:53:19.015411 11648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: No bootstrap required, opened a new log
I20260928 17:53:19.017486 11648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 17:53:19.017652 11648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:53:19.017712 11648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f4b3908ef04ab796d86e6a4f1d4156, State: Initialized, Role: FOLLOWER
I20260928 17:53:19.017855 11648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 17:53:19.017959 11648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:53:19.018013 11648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:53:19.018079 11648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:53:19.018608 11648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 17:53:19.018728 11648 leader_election.cc:302] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f4b3908ef04ab796d86e6a4f1d4156; no voters:
I20260928 17:53:19.018893 11648 leader_election.cc:288] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:53:19.018939 11653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:53:19.019137 11653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [term 1 LEADER]: Becoming Leader. State: Replica: 14f4b3908ef04ab796d86e6a4f1d4156, State: Running, Role: LEADER
I20260928 17:53:19.019240 11648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:53:19.019284 11653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } }
I20260928 17:53:19.019676 11653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 17:53:19.019845 11653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [sys.catalog]: This master's current role is: LEADER
I20260928 17:53:19.020032 11653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14f4b3908ef04ab796d86e6a4f1d4156. Latest consensus state: current_term: 1 leader_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f4b3908ef04ab796d86e6a4f1d4156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40235 } } }
I20260928 17:53:19.020187 11653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156 [sys.catalog]: This master's current role is: LEADER
I20260928 17:53:19.020357 11662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:53:19.020953 11662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:53:19.021226 11648 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 17:53:19 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:53:19.026679 11670 mvcc.cc:205] Tried to move back new op lower bound from 7334371324007677952 to 7334371323983585280. Current Snapshot: MvccSnapshot[applied={T|T < 7334371324007677952}]
I20260928 17:53:19.027756 11662 catalog_manager.cc:1347] Generated new cluster ID: ee273cd9b1e84d4ca23cd9bd71af6679
I20260928 17:53:19.027848 11662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:53:19.030655 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:19.015141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.98.62:40235)
0928 17:53:19.015318 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:19.015329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:19.015363 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:53:19.015910 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:19.015919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:19.015984 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:19.016167 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:53:19.016938 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:19.018654 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:19.019693 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:19.020356 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:19.020452 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:19.024300 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:53:19.024362 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:53:19.024365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:53:19.024474 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:53:19.027859 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:19.028313 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:19.028316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:19.028317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:19.028416 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:19.028637 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:19.028637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:19.028638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:19.029084 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:53:19.029255 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:19.029810 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 17:53:19.030151 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 17:53:19.040537 11662 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:53:19.041412 11662 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:53:19.053135 11662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: Generated new TSK 0
I20260928 17:53:19.053442 11662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:53:19.828049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:53:19.917629 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:53:19.917836 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:53:19.917860 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:53:19.919157 11693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:53:19.919202 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:53:19.919230 11693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:53:19.920976 11693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:53:19.921213 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:53:19.921473 11693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:53:19.926681 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:19.926808 11699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:19.926728 11701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:53:19.927662 11693 server_base.cc:1034] running on GCE node
I20260928 17:53:19.927876 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:53:19.928181 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:53:19.929337 11693 hybrid_clock.cc:648] HybridClock initialized: now 1790617999929327 us; error 28 us; skew 500 ppm
Sep 28 17:53:19 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:53:19.932343 11693 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:53:19.932916 11693 webserver.cc:460] Webserver started at http://127.0.98.1:37719/ using document root <none> and password file <none>
I20260928 17:53:19.933174 11693 fs_manager.cc:359] Metadata directory not provided
I20260928 17:53:19.933219 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:53:19.933346 11693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:53:19 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:53:19.950243 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "7dbce451c58e45208f0ac0a0eb613de9"
format_stamp: "Formatted at 2026-09-28 17:53:19 on dist-test-slave-zqlt"
server_key: "7ca13b43b708741b4ea51a54fcb4677c"
server_key_iv: "3b5a2e77deb9a5a54ec31e664df85cda"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:19.950719 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7dbce451c58e45208f0ac0a0eb613de9"
format_stamp: "Formatted at 2026-09-28 17:53:19 on dist-test-slave-zqlt"
server_key: "7ca13b43b708741b4ea51a54fcb4677c"
server_key_iv: "3b5a2e77deb9a5a54ec31e664df85cda"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:19.954888 11693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 17:53:19.972265 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:19.972541 11693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 17:53:19.972705 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "7dbce451c58e45208f0ac0a0eb613de9"
format_stamp: "Formatted at 2026-09-28 17:53:19 on dist-test-slave-zqlt"
server_key: "7ca13b43b708741b4ea51a54fcb4677c"
server_key_iv: "3b5a2e77deb9a5a54ec31e664df85cda"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:19.972824 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:53:19.999614 11693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:53:20.002149 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:53:20.002324 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:53:20.002589 11693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:53:20.002959 11693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:53:20.003010 11693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.003084 11693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:53:20.003127 11693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.009783 11693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38349
I20260928 17:53:20.010304 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:53:20.016961 11823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38349 every 8 connection(s)
I20260928 17:53:20.019408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 11693
I20260928 17:53:20.019515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:53:20.019723 392 external_mini_cluster.cc:1428] Setting key 568b11699d225e31648f307ed69e4d56
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790617999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:53:20.036389 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.017735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36635 (local address 127.0.98.62:40235)
0928 17:53:20.018838 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.018840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.018848 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:53:20.018873 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.018874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.018885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.019007 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:53:20.019838 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.020661 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.022172 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.022466 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.022536 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.032537 (+ 10001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:53:20.032538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:53:20.032539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:53:20.032547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:53:20.034486 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.035003 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.035004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.035005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.035068 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.035343 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.035344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.035344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.035502 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:53:20.035565 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.036152 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.036286 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":95,"threads_started":1}
I20260928 17:53:20.036928 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.017712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.1:36635)
0928 17:53:20.017953 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.018008 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.018642 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.019148 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.019153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.019421 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:53:20.019722 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.019728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.020756 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.020757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.022070 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.022073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.022607 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.022608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.022750 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.030664 (+ 7914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:53:20.030676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:53:20.032398 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:53:20.034593 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.034601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.034604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.034914 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.035134 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.035134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.035135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.035276 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.035595 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:53:20.035603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:53:20.035794 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.035907 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.036069 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260928 17:53:20.039556 11824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 17:53:20.039706 11824 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 17:53:20.039917 11824 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 17:53:20.040555 11612 ts_manager.cc:194] Registered new tserver with Master: 7dbce451c58e45208f0ac0a0eb613de9 (127.0.98.1:38349)
I20260928 17:53:20.041477 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:53:20.056383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:53:20.180565 11830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:53:20.180925 11830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:53:20.181010 11830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:53:20.182874 11830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:53:20.183118 11830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:53:20.183214 11830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:53:20.185559 11830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:53:20.186057 11830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:53:20.186396 11830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:53:20.193200 11838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:20.193593 11837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:20.193840 11840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:53:20.194772 11830 server_base.cc:1034] running on GCE node
I20260928 17:53:20.194995 11830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:53:20.195281 11830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:53:20.196473 11830 hybrid_clock.cc:648] HybridClock initialized: now 1790618000196451 us; error 36 us; skew 500 ppm
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:53:20.200016 11830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:53:20.200610 11830 webserver.cc:460] Webserver started at http://127.0.98.2:41427/ using document root <none> and password file <none>
I20260928 17:53:20.200888 11830 fs_manager.cc:359] Metadata directory not provided
I20260928 17:53:20.200944 11830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:53:20.201076 11830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:53:20.215677 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "237f7ab19cf149bfa4ad092abae4fe42"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "0e26eca1492fcc6a82009e66cf1f3399"
server_key_iv: "728226865e473512c2b27edc4163b0dd"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.216611 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "237f7ab19cf149bfa4ad092abae4fe42"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "0e26eca1492fcc6a82009e66cf1f3399"
server_key_iv: "728226865e473512c2b27edc4163b0dd"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.224581 11830 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:53:20.240264 11847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.240545 11830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 17:53:20.240675 11830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "237f7ab19cf149bfa4ad092abae4fe42"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "0e26eca1492fcc6a82009e66cf1f3399"
server_key_iv: "728226865e473512c2b27edc4163b0dd"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.240867 11830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:53:20.258380 11830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:53:20.260938 11830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:53:20.261147 11830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:53:20.261408 11830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:53:20.261840 11830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:53:20.261927 11830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.262005 11830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:53:20.262063 11830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.275550 11830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40673
I20260928 17:53:20.276186 11830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:53:20.276798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 11830
I20260928 17:53:20.276919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:53:20.277140 392 external_mini_cluster.cc:1428] Setting key 240cc68b6305e640a82ab44ce53519b3
I20260928 17:53:20.277295 11960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40673 every 8 connection(s)
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:53:20.288748 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.278280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47355 (local address 127.0.98.62:40235)
0928 17:53:20.278324 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.278326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.278529 (+ 203us) server_negotiation.cc:407] Connection header received
0928 17:53:20.279329 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.279331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.279344 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.279378 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:53:20.280308 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.280952 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.281946 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.282271 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.282340 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.284948 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:53:20.284949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:53:20.284950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:53:20.284958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:53:20.286899 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.287429 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.287430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.287430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.287505 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.287815 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.287815 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.287816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.287962 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:53:20.288009 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.288519 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.288616 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:53:20.289602 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.278005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.2:47355)
0928 17:53:20.278390 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.278438 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.279204 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.279601 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.279607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.279876 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:53:20.280201 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.280207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.281043 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.281043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.281845 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.281847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.282403 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.282404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.282548 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.283141 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:53:20.283151 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:53:20.284842 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:53:20.287006 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.287012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.287015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.287335 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.287585 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.287586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.287586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.287738 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.288059 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:53:20.288066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:53:20.288291 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.289131 (+ 840us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.289311 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":89,"threads_started":1}
I20260928 17:53:20.290293 11961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 17:53:20.290393 11961 heartbeater.cc:464] Registering TS with master...
I20260928 17:53:20.290561 11961 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 17:53:20.290902 11612 ts_manager.cc:194] Registered new tserver with Master: 237f7ab19cf149bfa4ad092abae4fe42 (127.0.98.2:40673)
I20260928 17:53:20.291335 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:53:20.306761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:53:20.425604 11966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:53:20.425980 11966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:53:20.426069 11966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:53:20.427873 11966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:53:20.428162 11966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:53:20.428277 11966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:53:20.430428 11966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:53:20.430933 11966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:53:20.431258 11966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:53:20.437786 11973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:20.438181 11972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:53:20.438448 11975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:53:20.449393 11966 server_base.cc:1034] running on GCE node
I20260928 17:53:20.449852 11966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:53:20.450230 11966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:53:20.457544 11966 hybrid_clock.cc:648] HybridClock initialized: now 1790618000457511 us; error 42 us; skew 500 ppm
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:53:20.461923 11966 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:53:20.462574 11966 webserver.cc:460] Webserver started at http://127.0.98.3:39183/ using document root <none> and password file <none>
I20260928 17:53:20.462909 11966 fs_manager.cc:359] Metadata directory not provided
I20260928 17:53:20.463049 11966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:53:20.463253 11966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:53:20.481796 11966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0a7bccfdd0d4445825aa4dc862bd12c"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "1afbb9983b63a91093f6a383ec07120b"
server_key_iv: "5801fa05dee55b9234b94233560b80a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.482380 11966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0a7bccfdd0d4445825aa4dc862bd12c"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "1afbb9983b63a91093f6a383ec07120b"
server_key_iv: "5801fa05dee55b9234b94233560b80a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.485236 11966 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 17:53:20.501647 11982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.501904 11966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 17:53:20.502015 11966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "b0a7bccfdd0d4445825aa4dc862bd12c"
format_stamp: "Formatted at 2026-09-28 17:53:20 on dist-test-slave-zqlt"
server_key: "1afbb9983b63a91093f6a383ec07120b"
server_key_iv: "5801fa05dee55b9234b94233560b80a3"
server_key_version: "kuduclusterkey@0"
I20260928 17:53:20.502112 11966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:53:20.524852 11966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:53:20.527410 11966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:53:20.527633 11966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:53:20.527989 11966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:53:20.528421 11966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:53:20.528523 11966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.528599 11966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:53:20.528666 11966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:53:20.535607 11966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32983
I20260928 17:53:20.536165 11966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:53:20.539053 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32983 every 8 connection(s)
I20260928 17:53:20.540818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 11966
I20260928 17:53:20.540941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:53:20.541154 392 external_mini_cluster.cc:1428] Setting key 30d193b21149833ab9dc89a9c62d3821
Sep 28 17:53:20 dist-test-slave-zqlt krb5kdc[11136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:53:20.553792 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.540262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60891 (local address 127.0.98.62:40235)
0928 17:53:20.540308 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.540310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.540550 (+ 240us) server_negotiation.cc:407] Connection header received
0928 17:53:20.541642 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.541642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.541656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.541686 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:53:20.545569 (+ 3883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.546356 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.547871 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.548205 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.548294 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.550695 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:53:20.550697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:53:20.550698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:53:20.550706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:53:20.552308 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.552793 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.552794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.552794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.552852 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.553133 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.553134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.553134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.553253 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:53:20.553328 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.553618 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.553700 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:53:20.554138 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.539945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40235 (local address 127.0.98.3:60891)
0928 17:53:20.540394 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.540458 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.541322 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.542135 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.542139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.542390 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:53:20.545425 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.545434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.546496 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.546497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.547763 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.547767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.548381 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.548382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.548538 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.549155 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:53:20.549173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:53:20.550593 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:53:20.552400 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.552407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.552410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.552708 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.552929 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:53:20.552930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:53:20.552930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:53:20.553062 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:53:20.553320 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:53:20.553326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:53:20.553480 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.553595 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.553740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":88,"threads_started":1}
I20260928 17:53:20.554550 12096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40235
I20260928 17:53:20.554642 12096 heartbeater.cc:464] Registering TS with master...
I20260928 17:53:20.554823 12096 heartbeater.cc:511] Master 127.0.98.62:40235 requested a full tablet report, sending...
I20260928 17:53:20.555250 11612 ts_manager.cc:194] Registered new tserver with Master: b0a7bccfdd0d4445825aa4dc862bd12c (127.0.98.3:32983)
I20260928 17:53:20.555722 11612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60891
I20260928 17:53:20.564433 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:53:20.577308 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.568561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.0.98.62:40235)
0928 17:53:20.568626 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.568628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.568784 (+ 156us) server_negotiation.cc:407] Connection header received
0928 17:53:20.568872 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.568873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.568884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.568951 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:53:20.569769 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.570487 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.571614 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.572320 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.572398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.574089 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:53:20.574090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:53:20.574091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:53:20.574098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:53:20.575913 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.576445 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.576447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.576447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.576512 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:53:20.576747 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:53:20.576748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:53:20.576748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:53:20.576875 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:53:20.576971 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.577140 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.577217 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:53:20.579487 11612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:53:20.582162 11612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:53:20.606019 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.586052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38349 (local address 127.0.0.1:43982)
0928 17:53:20.596470 (+ 10418us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.596562 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.601112 (+ 4550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.601397 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.601401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.601404 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:53:20.601505 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.601507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.602859 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.602860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.604377 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.604380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.605682 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.605683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.605766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.605780 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.605848 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.605924 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10363,"thread_start_us":107,"threads_started":1}
I20260928 17:53:20.606527 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.586675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.0.98.1:38349)
0928 17:53:20.586838 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.586843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.600970 (+ 14127us) server_negotiation.cc:407] Connection header received
0928 17:53:20.601265 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.601269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.601288 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.601373 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:53:20.601615 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.602757 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.604497 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.605513 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.605586 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.605650 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.606408 (+ 758us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.606453 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 17:53:20.607625 11758 tablet_service.cc:1467] Processing CreateTablet for tablet 9d656f7a1e46454bab0446389d0ab20e (DEFAULT_TABLE table=test-table [id=40cc7f1c4a034e819133c18a126701db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:53:20.608045 11758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d656f7a1e46454bab0446389d0ab20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:53:20.609921 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.586534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32983 (local address 127.0.0.1:50508)
0928 17:53:20.606842 (+ 20308us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.606865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.606915 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.607215 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.607216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.607218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:53:20.607252 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.607253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.608251 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.608252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.608951 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.608953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.609689 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.609690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.609746 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.609750 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.609807 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.609848 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20291,"thread_start_us":66,"threads_started":1}
I20260928 17:53:20.609943 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.587008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.0.98.3:32983)
0928 17:53:20.587259 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.587263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.606982 (+ 19719us) server_negotiation.cc:407] Connection header received
0928 17:53:20.607033 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.607035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.607054 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.607127 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:53:20.607375 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.608181 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.609069 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.609571 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.609685 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.609768 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.609824 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.609869 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260928 17:53:20.610899 12030 tablet_service.cc:1467] Processing CreateTablet for tablet 9d656f7a1e46454bab0446389d0ab20e (DEFAULT_TABLE table=test-table [id=40cc7f1c4a034e819133c18a126701db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:53:20.611246 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d656f7a1e46454bab0446389d0ab20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:53:20.611804 12115 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: Bootstrap starting.
I20260928 17:53:20.612666 12115 tablet_bootstrap.cc:654] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: Neither blocks nor log segments found. Creating new log.
I20260928 17:53:20.612751 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.586472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.0.98.2:40673)
0928 17:53:20.587188 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.587193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.601593 (+ 14400us) server_negotiation.cc:407] Connection header received
0928 17:53:20.601728 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.601731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.601749 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.601809 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:53:20.602090 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.603238 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.611358 (+ 8120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.612214 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.612274 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.612353 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.612618 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.612662 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":88,"threads_started":1}
I20260928 17:53:20.613008 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.586328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40673 (local address 127.0.0.1:44126)
0928 17:53:20.601509 (+ 15181us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.601522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.601624 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.601944 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.601945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.601947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:53:20.602003 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.602005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.603358 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.603359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.604531 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.604534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.612430 (+ 7896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.612431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.612506 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.612510 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.612863 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.612923 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15158,"thread_start_us":70,"threads_started":1}
I20260928 17:53:20.613381 12115 log.cc:824] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:53:20.614214 12115 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: No bootstrap required, opened a new log
I20260928 17:53:20.614285 12115 ts_tablet_manager.cc:1397] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:53:20.616130 11895 tablet_service.cc:1467] Processing CreateTablet for tablet 9d656f7a1e46454bab0446389d0ab20e (DEFAULT_TABLE table=test-table [id=40cc7f1c4a034e819133c18a126701db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:53:20.616506 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d656f7a1e46454bab0446389d0ab20e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:53:20.617303 12115 raft_consensus.cc:348] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.617466 12115 raft_consensus.cc:374] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:53:20.617498 12115 raft_consensus.cc:729] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbce451c58e45208f0ac0a0eb613de9, State: Initialized, Role: FOLLOWER
I20260928 17:53:20.618232 12115 consensus_queue.cc:260] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.618646 12115 ts_tablet_manager.cc:1428] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 17:53:20.618837 11824 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 17:53:20.619001 12117 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Bootstrap starting.
I20260928 17:53:20.620213 12117 tablet_bootstrap.cc:654] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Neither blocks nor log segments found. Creating new log.
I20260928 17:53:20.620649 12117 log.cc:824] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Log is configured to *not* fsync() on all Append() calls
I20260928 17:53:20.621451 12117 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: No bootstrap required, opened a new log
I20260928 17:53:20.621515 12117 ts_tablet_manager.cc:1397] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 17:53:20.623462 12117 raft_consensus.cc:348] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.623629 12117 raft_consensus.cc:374] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:53:20.623661 12117 raft_consensus.cc:729] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a7bccfdd0d4445825aa4dc862bd12c, State: Initialized, Role: FOLLOWER
I20260928 17:53:20.623769 12117 consensus_queue.cc:260] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.624020 12117 ts_tablet_manager.cc:1428] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:53:20.624243 12096 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 17:53:20.631767 12119 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: Bootstrap starting.
I20260928 17:53:20.633205 12119 tablet_bootstrap.cc:654] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: Neither blocks nor log segments found. Creating new log.
I20260928 17:53:20.633646 12119 log.cc:824] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: Log is configured to *not* fsync() on all Append() calls
I20260928 17:53:20.634600 12119 tablet_bootstrap.cc:492] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: No bootstrap required, opened a new log
I20260928 17:53:20.634668 12119 ts_tablet_manager.cc:1397] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:53:20.637830 12119 raft_consensus.cc:348] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.638100 12119 raft_consensus.cc:374] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:53:20.638180 12119 raft_consensus.cc:729] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f7ab19cf149bfa4ad092abae4fe42, State: Initialized, Role: FOLLOWER
I20260928 17:53:20.638372 12119 consensus_queue.cc:260] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.638711 12119 ts_tablet_manager.cc:1428] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:53:20.638904 11961 heartbeater.cc:503] Master 127.0.98.62:40235 was elected leader, sending a full tablet report...
I20260928 17:53:20.660331 12123 raft_consensus.cc:482] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:53:20.660614 12123 raft_consensus.cc:504] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.665695 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.661328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38349 (local address 127.0.98.2:47443)
0928 17:53:20.661442 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.661455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.661504 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.661838 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.661839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.661841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:53:20.661914 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.661916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.662861 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.662863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.664583 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.664586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.665426 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.665427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.665503 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.665507 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.665569 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.665616 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 17:53:20.666251 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.661610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47443 (local address 127.0.98.1:38349)
0928 17:53:20.661682 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.661684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.661693 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:53:20.661722 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.661723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.661734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.661785 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:53:20.662004 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.662756 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.664678 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.665302 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.665382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.665394 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.666135 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.666179 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 17:53:20.666554 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d656f7a1e46454bab0446389d0ab20e" candidate_uuid: "237f7ab19cf149bfa4ad092abae4fe42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbce451c58e45208f0ac0a0eb613de9" is_pre_election: true
I20260928 17:53:20.666744 11763 raft_consensus.cc:2454] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f7ab19cf149bfa4ad092abae4fe42 in term 0.
I20260928 17:53:20.667045 11849 leader_election.cc:302] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f7ab19cf149bfa4ad092abae4fe42, 7dbce451c58e45208f0ac0a0eb613de9; no voters:
I20260928 17:53:20.670616 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.666749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32983 (local address 127.0.98.2:38339)
0928 17:53:20.667052 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.667063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.667111 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.667437 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:53:20.667438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:53:20.667440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:53:20.667481 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.667483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.668426 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.668427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.669620 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:53:20.669624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.670401 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.670402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:53:20.670464 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.670467 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:53:20.670513 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:53:20.670547 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260928 17:53:20.670948 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.666858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38339 (local address 127.0.98.3:32983)
0928 17:53:20.666903 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.666905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.667163 (+ 258us) server_negotiation.cc:407] Connection header received
0928 17:53:20.667191 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.667192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.667204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.667246 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:53:20.667572 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.668310 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.669734 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.670286 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.670357 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.670368 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.670846 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.670885 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:53:20.671469 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d656f7a1e46454bab0446389d0ab20e" candidate_uuid: "237f7ab19cf149bfa4ad092abae4fe42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" is_pre_election: true
I20260928 17:53:20.671630 12046 raft_consensus.cc:2454] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f7ab19cf149bfa4ad092abae4fe42 in term 0.
I20260928 17:53:20.671914 12123 leader_election.cc:288] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dbce451c58e45208f0ac0a0eb613de9 (127.0.98.1:38349), b0a7bccfdd0d4445825aa4dc862bd12c (127.0.98.3:32983)
I20260928 17:53:20.672101 12123 raft_consensus.cc:2781] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:53:20.672206 12123 raft_consensus.cc:482] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:53:20.672277 12123 raft_consensus.cc:3037] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:53:20.673260 12123 raft_consensus.cc:504] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.675935 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d656f7a1e46454bab0446389d0ab20e" candidate_uuid: "237f7ab19cf149bfa4ad092abae4fe42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbce451c58e45208f0ac0a0eb613de9"
I20260928 17:53:20.676038 11763 raft_consensus.cc:3037] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:53:20.676993 11763 raft_consensus.cc:2454] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f7ab19cf149bfa4ad092abae4fe42 in term 1.
I20260928 17:53:20.677289 11849 leader_election.cc:302] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f7ab19cf149bfa4ad092abae4fe42, 7dbce451c58e45208f0ac0a0eb613de9; no voters:
I20260928 17:53:20.677403 12124 raft_consensus.cc:2781] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:53:20.677601 12124 raft_consensus.cc:686] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 1 LEADER]: Becoming Leader. State: Replica: 237f7ab19cf149bfa4ad092abae4fe42, State: Running, Role: LEADER
I20260928 17:53:20.677582 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d656f7a1e46454bab0446389d0ab20e" candidate_uuid: "237f7ab19cf149bfa4ad092abae4fe42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c"
I20260928 17:53:20.677660 12046 raft_consensus.cc:3037] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:53:20.677702 12124 consensus_queue.cc:237] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } }
I20260928 17:53:20.678484 12046 raft_consensus.cc:2454] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f7ab19cf149bfa4ad092abae4fe42 in term 1.
I20260928 17:53:20.678606 11612 catalog_manager.cc:5495] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 237f7ab19cf149bfa4ad092abae4fe42 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "237f7ab19cf149bfa4ad092abae4fe42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237f7ab19cf149bfa4ad092abae4fe42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } health_report { overall_health: UNKNOWN } } }
I20260928 17:53:20.678839 12123 leader_election.cc:288] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [CANDIDATE]: Term 1 election: Requested vote from peers 7dbce451c58e45208f0ac0a0eb613de9 (127.0.98.1:38349), b0a7bccfdd0d4445825aa4dc862bd12c (127.0.98.3:32983)
I20260928 17:53:20.688772 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.685437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.98.2:40673)
0928 17:53:20.685501 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:53:20.685504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:53:20.685512 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:53:20.685545 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:53:20.685545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:53:20.685563 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:53:20.685630 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:53:20.685831 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.686586 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.687698 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:53:20.687995 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:53:20.688059 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:53:20.688270 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:53:20.688391 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:53:20.688458 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:53:20.688666 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 17:53:20.688685 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 17:53:20.690857 12046 raft_consensus.cc:1261] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 1 FOLLOWER]: Refusing update from remote peer 237f7ab19cf149bfa4ad092abae4fe42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:53:20.691217 12125 consensus_queue.cc:1035] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a7bccfdd0d4445825aa4dc862bd12c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:53:20.693300 11763 raft_consensus.cc:1261] T 9d656f7a1e46454bab0446389d0ab20e P 7dbce451c58e45208f0ac0a0eb613de9 [term 1 FOLLOWER]: Refusing update from remote peer 237f7ab19cf149bfa4ad092abae4fe42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:53:20.693610 12125 consensus_queue.cc:1035] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dbce451c58e45208f0ac0a0eb613de9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:53:20.696893 12128 mvcc.cc:205] Tried to move back new op lower bound from 7334371330826944512 to 7334371330776584192. Current Snapshot: MvccSnapshot[applied={T|T < 7334371330826944512}]
I20260928 17:53:20.697845 12131 mvcc.cc:205] Tried to move back new op lower bound from 7334371330826944512 to 7334371330776584192. Current Snapshot: MvccSnapshot[applied={T|T < 7334371330826944512}]
I20260928 17:53:20.699648 12129 mvcc.cc:205] Tried to move back new op lower bound from 7334371330826944512 to 7334371330776584192. Current Snapshot: MvccSnapshot[applied={T|T < 7334371330826944512}]
I20260928 17:53:20.702625 11612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:53:20.702706 11612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:53:20.703408 11612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14f4b3908ef04ab796d86e6a4f1d4156: Sending DeleteTablet for 3 replicas of tablet 9d656f7a1e46454bab0446389d0ab20e
I20260928 17:53:20.703876 11758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d656f7a1e46454bab0446389d0ab20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43982
I20260928 17:53:20.703887 12030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d656f7a1e46454bab0446389d0ab20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50508
I20260928 17:53:20.704130 11895 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d656f7a1e46454bab0446389d0ab20e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44126
I20260928 17:53:20.704734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 11693
I20260928 17:53:20.715682 12139 tablet_replica.cc:330] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42: stopping tablet replica
I20260928 17:53:20.715829 12139 raft_consensus.cc:2229] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:53:20.715986 12139 raft_consensus.cc:2258] T 9d656f7a1e46454bab0446389d0ab20e P 237f7ab19cf149bfa4ad092abae4fe42 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 17:53:20.716342 11597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38349 (error 108)
I20260928 17:53:20.716486 12138 tablet_replica.cc:330] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: stopping tablet replica
I20260928 17:53:20.716614 12138 raft_consensus.cc:2229] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:53:20.716741 12138 raft_consensus.cc:2258] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:53:20.716959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 11830
I20260928 17:53:20.718605 12138 ts_tablet_manager.cc:1905] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:53:20.720861 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:53:20.720667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38349 (local address 127.0.0.1:43998)
0928 17:53:20.720751 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.720803 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 17:53:20.721041 11597 catalog_manager.cc:4584] TS 7dbce451c58e45208f0ac0a0eb613de9 (127.0.98.1:38349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d656f7a1e46454bab0446389d0ab20e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38349: connect: Connection refused (error 111)
I20260928 17:53:20.721576 12138 ts_tablet_manager.cc:1918] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:53:20.721654 12138 log.cc:1196] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/9d656f7a1e46454bab0446389d0ab20e
I20260928 17:53:20.722263 12138 ts_tablet_manager.cc:1939] T 9d656f7a1e46454bab0446389d0ab20e P b0a7bccfdd0d4445825aa4dc862bd12c: Deleting consensus metadata
I20260928 17:53:20.722726 11599 catalog_manager.cc:4841] TS b0a7bccfdd0d4445825aa4dc862bd12c (127.0.98.3:32983): tablet 9d656f7a1e46454bab0446389d0ab20e (table test-table [id=40cc7f1c4a034e819133c18a126701db]) successfully deleted
W20260928 17:53:20.725124 12110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:53:20.724846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40673 (local address 127.0.0.1:44144)
0928 17:53:20.724890 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:53:20.724901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:53:20.724953 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:53:20.725079 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:53:20.725571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 11966
I20260928 17:53:20.730396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 11580
I20260928 17:53:20.736963 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 17:53:21.071666 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 17:53:21.071766 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 17:53:21.416625 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 17:53:21.416 UTC [11166] LOG: received smart shutdown request
2026-09-28 17:53:21.419 UTC [11166] LOG: background worker "logical replication launcher" (PID 11175) exited with exit code 1
2026-09-28 17:53:21.420 UTC [11170] LOG: shutting down
2026-09-28 17:53:21.586 UTC [11166] LOG: database system is shut down
2026-09-28T17:53:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46938 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:53:21 dist-test-slave-zqlt krb5kdc[12149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:53:21 dist-test-slave-zqlt krb5kdc[12149](info): set up 2 sockets
Sep 28 17:53:21 dist-test-slave-zqlt krb5kdc[12149](info): commencing operation
krb5kdc: starting...
W20260928 17:53:22.728214 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:53:22 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:53:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 17:53:22.890043 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMA1V5K/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/postgres -l logfile start
2026-09-28 17:53:25.364 UTC [12179] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 17:53:25.364 UTC [12179] LOG: listening on IPv4 address "127.0.98.20", port 37275
2026-09-28 17:53:25.368 UTC [12179] LOG: listening on Unix socket "/tmp/.s.PGSQL.37275"
2026-09-28 17:53:25.388 UTC [12182] LOG: database system was shut down at 2026-09-28 17:53:23 UTC
2026-09-28 17:53:25.392 UTC [12179] LOG: database system is ready to accept connections
I20260928 17:53:26.348546 392 mini_postgres.cc:96] Postgres bound to 37275
2026-09-28 17:53:26.355 UTC [12190] FATAL: database "slave" does not exist
127.0.98.20:37275 - accepting connections
I20260928 17:53:26.355818 392 mini_ranger.cc:161] Starting Ranger...
I20260928 17:53:26.371824 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 17:53:26.896849 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 17:53:26.896919 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin
2026-09-28 17:53:27,187 [I] DB FLAVOR :POSTGRES
2026-09-28 17:53:27,188 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:53:27,188 [I] Checking connection..
2026-09-28 17:53:27,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 17:53:27,445 [I] Checking connection passed.
2026-09-28 17:53:27,446 [I] --------- Verifying version history table ---------
2026-09-28 17:53:27,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:53:27,694 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:53:27,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:53:27,964 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 17:53:27,965 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 17:53:27,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 17:53:28.220 UTC [12254] WARNING: there is no transaction in progress
2026-09-28 17:53:28,226 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 17:53:28,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 17:53:28,472 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 17:53:28,472 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:53:28,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 17:53:28,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 17:53:28,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zqlt.c.gcp-upstream.internal','N') ;"
2026-09-28 17:53:29,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:53:29,461 [I] Table x_portal_user does not exist in database ranger
2026-09-28 17:53:29,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:53:29,725 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 17:53:29,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:53:29,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:53:30,224 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 17:53:30,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 17:53:31.200 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.217 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.234 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.249 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.266 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.284 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.338 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.345 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.359 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.370 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.380 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.395 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.404 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.410 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.417 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:31.424 UTC [12362] WARNING: there is no transaction in progress
2026-09-28 17:53:32,079 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 17:53:32,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 17:53:32,343 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 17:53:32,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 17:53:32,600 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 17:53:32,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 17:53:32,844 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 17:53:32,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 17:53:33,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 17:53:33,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zqlt.c.gcp-upstream.internal';"
2026-09-28 17:53:33,549 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 17:53:33,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 17:53:33,777 [I] DB_PATCHES have already been applied
I20260928 17:53:33.782986 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMA1V5K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 17:53:33.783058 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 17:53:33.786158 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48129
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 5:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 17:53:34.887652 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.532s user 0.000s sys 0.009s
Sep 28, 2026 5:53:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 5:53:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 5:53:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 5:53:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 5:53:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 17:53:57.113976 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 17:53:57 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618037, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 17:53:57.193694 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 17:53:57.202332 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 17:53:58.015938 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 17:53:58.016028 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms
I20260928 17:53:58.016045 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37275
2026-09-28 17:53:58,180 [I] DB FLAVOR :POSTGRES
2026-09-28 17:53:58,180 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 17:53:58,180 [I] Checking connection
2026-09-28 17:53:58,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 17:53:58,429 [I] connection success
2026-09-28 17:53:58,429 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 17:53:58,429 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 17:53:58,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 17:53:58,679 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 17:53:58,679 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 17:53:58,679 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 17:53:58,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 17:53:58,946 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 17:53:59.448935 392 mini_ranger_kms.cc:310] Created kms service
I20260928 17:53:59.648653 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 17:53:59.698947 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 17:53:59 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618037, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:00.131307 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 17:54:00.131546 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMA1V5K/build/release/bin/postgresql.jar:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMA1V5K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 17:54:00.131569 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 17:54:00.135408 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12542
I20260928 17:54:00.135506 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54147
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 5:54:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 17:54:02.439909 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.246s user 0.000s sys 0.146s
I20260928 17:54:02.440011 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 17:54:02.440056 392 mini_ranger_kms.cc:401] 127.0.98.20:54147/kms/v1/keys
Sep 28 17:54:02 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790618042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 17:54:03 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 17:54:03 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 17:54:05 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:54:05.764065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44813
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:05.876410 12593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:05.876641 12593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:05.876667 12593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:05.877892 12593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:05.877949 12593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:05.877966 12593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:05.877980 12593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:05.878032 12593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:05.879760 12593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44813
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:05.880024 12593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:05.880304 12593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:05.886598 12598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:05.886607 12599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:05.886605 12601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:05.886621 12593 server_base.cc:1034] running on GCE node
I20260928 17:54:05.887115 12593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:05.887477 12593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:05.888672 12593 hybrid_clock.cc:648] HybridClock initialized: now 1790618045888648 us; error 41 us; skew 500 ppm
Sep 28 17:54:05 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:05.895560 12593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:05.896130 12593 webserver.cc:460] Webserver started at http://127.0.98.62:45093/ using document root <none> and password file <none>
I20260928 17:54:05.896381 12593 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:05.896440 12593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:05.896592 12593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:54:05 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:05.941239 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "20165934d1b648acbfd8794c2d22f971"
format_stamp: "Formatted at 2026-09-28 17:54:05 on dist-test-slave-zqlt"
server_key: "1f28a4f04d99ea80235ea5f67625ea45"
server_key_iv: "f919f63c9595b513fb252cf4e4e6f189"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:05.941677 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "20165934d1b648acbfd8794c2d22f971"
format_stamp: "Formatted at 2026-09-28 17:54:05 on dist-test-slave-zqlt"
server_key: "1f28a4f04d99ea80235ea5f67625ea45"
server_key_iv: "f919f63c9595b513fb252cf4e4e6f189"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:05.943285 12593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:54:05.955849 12608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:05.956041 12593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:05.956172 12593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "20165934d1b648acbfd8794c2d22f971"
format_stamp: "Formatted at 2026-09-28 17:54:05 on dist-test-slave-zqlt"
server_key: "1f28a4f04d99ea80235ea5f67625ea45"
server_key_iv: "f919f63c9595b513fb252cf4e4e6f189"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:05.956297 12593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:05.979239 12593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:05.981613 12593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:05.981775 12593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:05.986871 12593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44813
I20260928 17:54:05.986950 12660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44813 every 8 connection(s)
I20260928 17:54:05.987347 12593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:05.988190 12661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:05.991314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 12593
I20260928 17:54:05.991345 12661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Bootstrap starting.
I20260928 17:54:05.991432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:54:05.991627 392 external_mini_cluster.cc:1428] Setting key 35028eda67b3c0aa09748fdc5c0fc06f
I20260928 17:54:05.992043 12661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:05.992450 12661 log.cc:824] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:05.993284 12661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: No bootstrap required, opened a new log
I20260928 17:54:05.995775 12661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:05.995965 12661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:05.995997 12661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20165934d1b648acbfd8794c2d22f971, State: Initialized, Role: FOLLOWER
I20260928 17:54:05.996198 12661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:05.996300 12661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:05.996341 12661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:05.996389 12661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:05.997073 12661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:05.997179 12661 leader_election.cc:302] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20165934d1b648acbfd8794c2d22f971; no voters:
I20260928 17:54:05.997371 12661 leader_election.cc:288] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:54:05.997430 12666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:05.997653 12666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 LEADER]: Becoming Leader. State: Replica: 20165934d1b648acbfd8794c2d22f971, State: Running, Role: LEADER
I20260928 17:54:05.997740 12661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:05.997804 12666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:05.998224 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20165934d1b648acbfd8794c2d22f971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
I20260928 17:54:05.998368 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:05.998548 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20165934d1b648acbfd8794c2d22f971. Latest consensus state: current_term: 1 leader_uuid: "20165934d1b648acbfd8794c2d22f971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
Sep 28 17:54:05 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:05.998654 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:05.998801 12675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:05.999380 12675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:05.999686 12661 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 17:54:06.010001 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:05.992202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.0.98.62:44813)
0928 17:54:05.992389 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:05.992399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:05.992435 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:54:05.993053 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:05.993061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:05.993116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:05.993281 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:05.993922 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:05.994821 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:05.995610 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:05.996129 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:05.996235 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:05.999254 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:05.999259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:05.999259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:05.999273 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:06.007868 (+ 8595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:06.008441 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:06.008444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:06.008444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:06.008561 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:06.008766 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:06.008766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:06.008766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:06.009108 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:06.009275 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:06.009589 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 17:54:06.009747 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260928 17:54:06.011827 12675 catalog_manager.cc:1347] Generated new cluster ID: 09f095a6cb8945b9b8eaa0bd51e547d3
I20260928 17:54:06.011893 12675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:06.028848 12675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:54:06.029688 12675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:06.032812 12675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Generated new TSK 0
I20260928 17:54:06.033118 12675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:54:06.781312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:06.870699 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:06.870911 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:06.870934 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:06.872188 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:06.872229 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:06.872256 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:06.873854 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:06.874136 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:06.874413 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:06.879465 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:06.879652 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:06.879475 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:06.880621 12706 server_base.cc:1034] running on GCE node
I20260928 17:54:06.880883 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:06.881125 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:06.884370 12706 hybrid_clock.cc:648] HybridClock initialized: now 1790618046884337 us; error 50 us; skew 500 ppm
Sep 28 17:54:06 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:06.887189 12706 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:06.887770 12706 webserver.cc:460] Webserver started at http://127.0.98.1:37229/ using document root <none> and password file <none>
I20260928 17:54:06.888048 12706 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:06.888149 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:06.888291 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:54:06 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:06.902196 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "31251564c0864c8586cdf54af5029910"
format_stamp: "Formatted at 2026-09-28 17:54:06 on dist-test-slave-zqlt"
server_key: "af7d51c5ef672c4aeacc8e33c13800c7"
server_key_iv: "1c06bc0f724ba5ade73b1c9a8461eed5"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:06.902740 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31251564c0864c8586cdf54af5029910"
format_stamp: "Formatted at 2026-09-28 17:54:06 on dist-test-slave-zqlt"
server_key: "af7d51c5ef672c4aeacc8e33c13800c7"
server_key_iv: "1c06bc0f724ba5ade73b1c9a8461eed5"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:06.905619 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 17:54:06.926015 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:06.926311 12706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:06.926437 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "31251564c0864c8586cdf54af5029910"
format_stamp: "Formatted at 2026-09-28 17:54:06 on dist-test-slave-zqlt"
server_key: "af7d51c5ef672c4aeacc8e33c13800c7"
server_key_iv: "1c06bc0f724ba5ade73b1c9a8461eed5"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:06.926544 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:06.943351 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:06.946085 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:06.946259 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:06.946518 12706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:06.946902 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:06.946943 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:06.946977 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:06.946996 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:06.953895 12706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41577
I20260928 17:54:06.954473 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:06.959193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 12706
I20260928 17:54:06.959297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:54:06.959497 392 external_mini_cluster.cc:1428] Setting key 85577befc54d0660c0e6a419eb122aed
I20260928 17:54:06.956456 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41577 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 28 17:54:06 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:06.982389 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:06.958018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.1:57213)
0928 17:54:06.961397 (+ 3379us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:06.961453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:06.962105 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:06.972278 (+ 10173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:06.972285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:06.972572 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:06.972901 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:06.972909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:06.974044 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:06.974045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:06.975363 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:06.975366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:06.975918 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:06.975919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:06.976062 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:06.976768 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:06.976779 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:06.978479 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:06.980625 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:06.980632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:06.980634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:06.980937 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:06.981170 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:06.981170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:06.981171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:06.981329 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:06.981653 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:06.981660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:06.981844 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 17:54:06.981954 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 17:54:06.982124 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3326,"thread_start_us":85,"threads_started":1}
I20260928 17:54:06.983078 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:06.958045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57213 (local address 127.0.98.62:44813)
0928 17:54:06.964222 (+ 6177us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:06.964225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:06.964235 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:06.964261 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:06.964262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:06.964275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:06.964354 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:06.973099 (+ 8745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:06.973931 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:06.975478 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:06.975766 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:06.975834 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:06.978603 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:06.978606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:06.978606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:06.978614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:06.980529 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:06.981030 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:06.981032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:06.981032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:06.981099 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:06.981411 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:06.981412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:06.981413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:06.981560 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:06.981623 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:06.982877 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0928 17:54:06.982968 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6143,"thread_start_us":90,"threads_started":1}
I20260928 17:54:06.984033 12837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:06.984158 12837 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:06.984366 12837 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:06.984906 12625 ts_manager.cc:194] Registered new tserver with Master: 31251564c0864c8586cdf54af5029910 (127.0.98.1:41577)
I20260928 17:54:06.985822 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:54:06.999192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:07.108772 12843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:07.109006 12843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:07.109047 12843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:07.110244 12843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:07.110316 12843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:07.110361 12843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:07.111948 12843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:07.112233 12843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:07.112475 12843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:07.117461 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.117476 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.117739 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.118571 12843 server_base.cc:1034] running on GCE node
I20260928 17:54:07.118770 12843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:07.119035 12843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:07.122231 12843 hybrid_clock.cc:648] HybridClock initialized: now 1790618047122209 us; error 33 us; skew 500 ppm
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:07.124846 12843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:07.125356 12843 webserver.cc:460] Webserver started at http://127.0.98.2:41439/ using document root <none> and password file <none>
I20260928 17:54:07.125612 12843 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:07.125680 12843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:07.125828 12843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:07.143122 12843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "de33a051644f4eb6a6ba77c54864bc67"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "265f31d912517fa66468409cc2595a04"
server_key_iv: "76a966ea39d4413f77b8b8d2e5e45562"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.143605 12843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de33a051644f4eb6a6ba77c54864bc67"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "265f31d912517fa66468409cc2595a04"
server_key_iv: "76a966ea39d4413f77b8b8d2e5e45562"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.146543 12843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:07.160895 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.161147 12843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:07.161337 12843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "de33a051644f4eb6a6ba77c54864bc67"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "265f31d912517fa66468409cc2595a04"
server_key_iv: "76a966ea39d4413f77b8b8d2e5e45562"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.161522 12843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:07.210062 12843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:07.212406 12843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:07.212576 12843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:07.212863 12843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:07.213282 12843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:07.213346 12843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.213410 12843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:07.213439 12843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.220763 12843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45895
I20260928 17:54:07.221165 12843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:54:07.221822 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45895 every 8 connection(s)
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:07.228629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 12843
I20260928 17:54:07.228739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:54:07.228953 392 external_mini_cluster.cc:1428] Setting key 0c751bf3387b558c4e426ab6e873702e
I20260928 17:54:07.231953 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.222282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48337 (local address 127.0.98.62:44813)
0928 17:54:07.222363 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:07.222365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:07.222381 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:54:07.222808 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:07.222810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:07.222822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:07.222869 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:07.223444 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.223956 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.225053 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.225276 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.225386 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.227932 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:07.227934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:07.227934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:07.227942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:07.229835 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.230718 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.230720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.230720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.230779 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.231090 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.231091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.231092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.231236 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:07.231277 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:07.231685 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 17:54:07.231801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:54:07.232764 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.221951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.2:48337)
0928 17:54:07.222122 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:07.222166 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:07.222694 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:07.222951 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:07.222955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:07.223142 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:07.223344 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.223348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.224068 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.224069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.224958 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.224960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.225408 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.225409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.225535 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.226128 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:07.226138 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:07.227809 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:07.230279 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.230287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.230289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.230624 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.230864 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.230864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.230865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.231017 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.231324 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:07.231330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:07.231530 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 17:54:07.232351 (+ 821us) client_negotiation.cc:240] Negotiation successful
0928 17:54:07.232523 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 17:54:07.233930 12974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:07.234053 12974 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:07.234306 12974 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:07.234740 12625 ts_manager.cc:194] Registered new tserver with Master: de33a051644f4eb6a6ba77c54864bc67 (127.0.98.2:45895)
I20260928 17:54:07.235270 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48337
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:54:07.269126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:07.401930 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:07.402276 12980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:07.402365 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:07.404230 12980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:07.404327 12980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:07.404405 12980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:07.406570 12980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:07.407132 12980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:07.407473 12980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:07.414325 12986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.414979 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.415156 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.416033 12980 server_base.cc:1034] running on GCE node
I20260928 17:54:07.416379 12980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:07.416663 12980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:07.417843 12980 hybrid_clock.cc:648] HybridClock initialized: now 1790618047417822 us; error 28 us; skew 500 ppm
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:07.421223 12980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:07.421775 12980 webserver.cc:460] Webserver started at http://127.0.98.3:36909/ using document root <none> and password file <none>
I20260928 17:54:07.422029 12980 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:07.422083 12980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:07.422212 12980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:07.441793 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "590157b62d814b84b030d205435d2284"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "ee715e49c448da150d6ecdf0d2989bc5"
server_key_iv: "c66347eeef9deda5f5d4c7bd360537ff"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.442375 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "590157b62d814b84b030d205435d2284"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "ee715e49c448da150d6ecdf0d2989bc5"
server_key_iv: "c66347eeef9deda5f5d4c7bd360537ff"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.445204 12980 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:07.467345 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.467607 12980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:07.467756 12980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "590157b62d814b84b030d205435d2284"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "ee715e49c448da150d6ecdf0d2989bc5"
server_key_iv: "c66347eeef9deda5f5d4c7bd360537ff"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.467936 12980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:07.493445 12980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:07.495919 12980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:07.496054 12980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:07.496270 12980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:07.496667 12980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:07.496721 12980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.496752 12980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:07.496809 12980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.504031 12980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37693
I20260928 17:54:07.504218 13108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37693 every 8 connection(s)
I20260928 17:54:07.504541 12980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:54:07.506018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 12980
I20260928 17:54:07.506129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:54:07.506316 392 external_mini_cluster.cc:1428] Setting key c45b7463ee62f03f2744e7daf8b2b1ef
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:07.516600 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.505319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34199 (local address 127.0.98.62:44813)
0928 17:54:07.505382 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:07.505385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:07.505699 (+ 314us) server_negotiation.cc:407] Connection header received
0928 17:54:07.506380 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:07.506382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:07.506396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:07.506432 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:07.507301 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.508114 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.509552 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.509871 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.509970 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.512756 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:07.512758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:07.512758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:07.512767 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:07.514629 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.515219 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.515220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.515220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.515290 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.515670 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.515671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.515672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.515825 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:07.515906 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:07.516399 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 17:54:07.516492 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:54:07.516886 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.505315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.3:34199)
0928 17:54:07.505523 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:07.505572 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:07.506241 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:07.506645 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:07.506649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:07.506905 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:07.507180 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.507187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.508243 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.508244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.509437 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.509440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.510089 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.510090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.510226 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.510887 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:07.510898 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:07.512631 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:07.514737 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.514745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.514749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.515117 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.515379 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.515381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.515381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.515572 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.515916 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:07.515924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:07.516216 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 17:54:07.516364 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 17:54:07.516567 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260928 17:54:07.517730 13109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:07.517867 13109 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:07.518088 13109 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:07.518572 12625 ts_manager.cc:194] Registered new tserver with Master: 590157b62d814b84b030d205435d2284 (127.0.98.3:37693)
I20260928 17:54:07.519145 12625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34199
I20260928 17:54:07.531314 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:07.532058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 12706
I20260928 17:54:07.540006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 12843
I20260928 17:54:07.546416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 12980
I20260928 17:54:07.552683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 12593
I20260928 17:54:07.558040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--webserver_interface=127.0.98.62
--webserver_port=45093
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44813
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:07.642791 13112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:07.643030 13112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:07.643069 13112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:07.644331 13112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:07.644392 13112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:07.644416 13112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:07.644436 13112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:07.644465 13112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:07.646116 13112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44813
--ranger_config_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:07.646400 13112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:07.646634 13112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:07.651737 13120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.651748 13117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.651851 13112 server_base.cc:1034] running on GCE node
W20260928 17:54:07.651975 13118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.652303 13112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:07.652608 13112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:07.653776 13112 hybrid_clock.cc:648] HybridClock initialized: now 1790618047653761 us; error 32 us; skew 500 ppm
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:07.656411 13112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:07.656864 13112 webserver.cc:460] Webserver started at http://127.0.98.62:45093/ using document root <none> and password file <none>
I20260928 17:54:07.657114 13112 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:07.657191 13112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:07.717206 13112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:07.717968 13128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.718240 13112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:07.718360 13112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "20165934d1b648acbfd8794c2d22f971"
format_stamp: "Formatted at 2026-09-28 17:54:05 on dist-test-slave-zqlt"
server_key: "1f28a4f04d99ea80235ea5f67625ea45"
server_key_iv: "f919f63c9595b513fb252cf4e4e6f189"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.718837 13112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:07.732832 13112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:07.734762 13112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:07.734884 13112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:07.739256 13112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44813
I20260928 17:54:07.739291 13180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44813 every 8 connection(s)
I20260928 17:54:07.739691 13112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:07.740996 13181 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:54:07.741678 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Bootstrap starting.
I20260928 17:54:07.744238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13112
I20260928 17:54:07.744571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41577
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44813
--webserver_port=37229
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:07.750406 13181 log.cc:824] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:07.752277 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:07.752580 13181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Bootstrap complete.
I20260928 17:54:07.754989 13181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:07.755237 13181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20165934d1b648acbfd8794c2d22f971, State: Initialized, Role: FOLLOWER
I20260928 17:54:07.755414 13181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:07.755487 13181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:07.755553 13181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:07.755599 13181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:54:07.756772 13181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:07.756889 13181 leader_election.cc:302] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20165934d1b648acbfd8794c2d22f971; no voters:
I20260928 17:54:07.757071 13181 leader_election.cc:288] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:54:07.757133 13185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:54:07.757316 13185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [term 2 LEADER]: Becoming Leader. State: Replica: 20165934d1b648acbfd8794c2d22f971, State: Running, Role: LEADER
I20260928 17:54:07.757357 13181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:07.757471 13185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } }
I20260928 17:54:07.758325 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20165934d1b648acbfd8794c2d22f971. Latest consensus state: current_term: 2 leader_uuid: "20165934d1b648acbfd8794c2d22f971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
I20260928 17:54:07.758440 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:07.758530 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20165934d1b648acbfd8794c2d22f971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20165934d1b648acbfd8794c2d22f971" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44813 } } }
I20260928 17:54:07.758605 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:07.758631 13193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:07.759231 13193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:07.759435 13193 catalog_manager.cc:1259] Loaded cluster ID: 09f095a6cb8945b9b8eaa0bd51e547d3
I20260928 17:54:07.759516 13193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:07.759639 13181 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 17:54:07.763098 13193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:07.763741 13193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Loaded TSK: 0
I20260928 17:54:07.764190 13193 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:54:07.862048 13183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:07.862462 13183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:07.862558 13183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:07.864603 13183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:07.864732 13183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:07.864835 13183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:07.867275 13183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37229
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:07.867799 13183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:07.868161 13183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:07.875082 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.875349 13183 server_base.cc:1034] running on GCE node
W20260928 17:54:07.875170 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:07.875660 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:07.875952 13183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:07.876257 13183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:07.877431 13183 hybrid_clock.cc:648] HybridClock initialized: now 1790618047877415 us; error 32 us; skew 500 ppm
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:07.880800 13183 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:07.881356 13183 webserver.cc:460] Webserver started at http://127.0.98.1:37229/ using document root <none> and password file <none>
I20260928 17:54:07.881626 13183 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:07.881747 13183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:07.905330 13183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:07.912251 13229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.912460 13183 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 17:54:07.912544 13183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "31251564c0864c8586cdf54af5029910"
format_stamp: "Formatted at 2026-09-28 17:54:06 on dist-test-slave-zqlt"
server_key: "af7d51c5ef672c4aeacc8e33c13800c7"
server_key_iv: "1c06bc0f724ba5ade73b1c9a8461eed5"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:07.913057 13183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:07.935608 13183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:07.937894 13183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:07.938068 13183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:07.938329 13183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:07.938649 13183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:07.938704 13183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.938740 13183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:07.938792 13183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:07.945083 13183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41577
I20260928 17:54:07.945181 13342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41577 every 8 connection(s)
I20260928 17:54:07.945538 13183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:07.951334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13183
I20260928 17:54:07.951645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45895
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44813
--webserver_port=41439
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:54:07 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:07.960352 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.946281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.1:39549)
0928 17:54:07.946498 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:07.946565 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:07.947214 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:07.947760 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:07.947764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:07.947982 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:07.948264 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.948270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.949752 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.949753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.951033 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:07.951036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.951990 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.951991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:07.952169 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.952795 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:07.952806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:07.954277 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:07.957154 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.957162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.957164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.958425 (+ 1261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.958748 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:07.958749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:07.958749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:07.958890 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:07.959435 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:07.959442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:07.959679 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 17:54:07.959893 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 17:54:07.960074 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":106,"threads_started":1}
I20260928 17:54:07.961304 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:07.946424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39549 (local address 127.0.98.62:44813)
0928 17:54:07.946652 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:07.946662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:07.946702 (+ 40us) server_negotiation.cc:407] Connection header received
0928 17:54:07.947486 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:07.947494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:07.947555 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:07.947731 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:07.948387 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.949436 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.951147 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:07.951743 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:07.951878 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:07.954401 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:07.954405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:07.954406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:07.954418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:07.956297 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.958532 (+ 2235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.958536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.958537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.958663 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:07.958971 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:07.958972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:07.958973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:07.959314 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:07.959402 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:07.960793 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0928 17:54:07.960927 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 17:54:07.962590 13343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:07.962719 13343 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:07.962972 13343 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:07.963953 13343 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:54:07.964027 13343 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:07.964111 13343 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:07.964457 13343 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:07.964527 13343 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:07.964867 13343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:54:08.070564 13347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:08.070932 13347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:08.071026 13347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:08.072857 13347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:08.072975 13347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:08.073065 13347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:08.075589 13347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41439
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:08.076023 13347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:08.076402 13347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:08.081472 13355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:08.082860 13352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:08.082988 13353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:08.083281 13347 server_base.cc:1034] running on GCE node
I20260928 17:54:08.083489 13347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:08.083776 13347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:08.084962 13347 hybrid_clock.cc:648] HybridClock initialized: now 1790618048084939 us; error 36 us; skew 500 ppm
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:08.087772 13347 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:08.088369 13347 webserver.cc:460] Webserver started at http://127.0.98.2:41439/ using document root <none> and password file <none>
I20260928 17:54:08.088608 13347 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:08.088691 13347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:08.102849 13347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:08.104171 13363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.104375 13347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:08.104496 13347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "de33a051644f4eb6a6ba77c54864bc67"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "265f31d912517fa66468409cc2595a04"
server_key_iv: "76a966ea39d4413f77b8b8d2e5e45562"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:08.105002 13347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:08.160848 13347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:08.163290 13347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:08.163484 13347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:08.163753 13347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:08.164130 13347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:08.164196 13347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.164237 13347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:08.164266 13347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.170652 13347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45895
I20260928 17:54:08.170727 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45895 every 8 connection(s)
I20260928 17:54:08.171072 13347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:08.179461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13347
I20260928 17:54:08.179757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37693
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44813
--webserver_port=36909
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:08.184324 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:08.171926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54625 (local address 127.0.98.62:44813)
0928 17:54:08.171979 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:08.171982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:08.172234 (+ 252us) server_negotiation.cc:407] Connection header received
0928 17:54:08.172760 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:08.172762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:08.172774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:08.172866 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:08.173446 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.174235 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.175644 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.175956 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.176034 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:08.178962 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:08.178963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:08.178964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:08.178973 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:08.182542 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.183081 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.183083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.183083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.183151 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.183472 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.183473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.183473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.183632 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:08.183680 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:08.184123 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 17:54:08.184220 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:54:08.184912 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:08.171901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.2:54625)
0928 17:54:08.172076 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:08.172149 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:08.172638 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:08.172929 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:08.172932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:08.173123 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:08.173334 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:08.173338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.174355 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.174356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:08.175539 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:08.175542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.176150 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.176152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:08.176325 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:08.177050 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:08.177062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:08.178840 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:08.182658 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:08.182665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:08.182668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:08.182982 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:08.183233 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:08.183234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:08.183235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:08.183383 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:08.183733 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:08.183740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:08.183936 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 17:54:08.184471 (+ 535us) client_negotiation.cc:240] Negotiation successful
0928 17:54:08.184665 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 17:54:08.185401 13477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:08.185511 13477 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.185722 13477 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:08.186334 13477 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:54:08.186410 13477 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.186476 13477 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:08.186777 13477 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.186853 13477 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:08.187111 13477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:54:08.299726 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:08.299917 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:08.299940 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:08.301144 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:08.301214 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:08.301263 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:08.302816 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:54147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36909
--tserver_master_addrs=127.0.98.62:44813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:08.303072 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:08.303304 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:08.309046 13489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:08.309093 13480 server_base.cc:1034] running on GCE node
W20260928 17:54:08.309170 13486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:08.309062 13487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:08.309623 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:08.309909 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:08.311082 13480 hybrid_clock.cc:648] HybridClock initialized: now 1790618048311060 us; error 40 us; skew 500 ppm
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:08.314060 13480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:08.314601 13480 webserver.cc:460] Webserver started at http://127.0.98.3:36909/ using document root <none> and password file <none>
I20260928 17:54:08.314854 13480 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:08.314936 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:08.331372 13480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:08.332181 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.332392 13480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:08.332490 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "590157b62d814b84b030d205435d2284"
format_stamp: "Formatted at 2026-09-28 17:54:07 on dist-test-slave-zqlt"
server_key: "ee715e49c448da150d6ecdf0d2989bc5"
server_key_iv: "c66347eeef9deda5f5d4c7bd360537ff"
server_key_version: "kuduclusterkey@0"
I20260928 17:54:08.332908 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:08.351591 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:08.353694 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:08.353853 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:08.354128 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:08.354502 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:08.354554 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.354595 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:08.354616 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:08.361791 13480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37693
I20260928 17:54:08.362272 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:54:08.362880 13610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37693 every 8 connection(s)
I20260928 17:54:08.364320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13480
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:08.374719 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:08.363682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36163 (local address 127.0.98.62:44813)
0928 17:54:08.363734 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:08.363736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:08.364020 (+ 284us) server_negotiation.cc:407] Connection header received
0928 17:54:08.364799 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:08.364801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:08.364814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:08.364848 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:08.365737 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.366522 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.368070 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.368409 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.368491 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:08.371196 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:08.371198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:08.371199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:08.371207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:08.373015 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.373546 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.373548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.373548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.373612 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.373912 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.373913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.373913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.374054 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:08.374101 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:08.374536 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 17:54:08.374649 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:54:08.375201 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:08.363690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44813 (local address 127.0.98.3:36163)
0928 17:54:08.363856 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:08.363909 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:08.364608 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:08.365050 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:08.365054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:08.365313 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:08.365615 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:08.365623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.366645 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.366646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:08.367960 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:08.367963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.368523 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.368525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:08.368674 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:08.369309 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:08.369319 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:08.371069 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:08.373115 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:08.373121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:08.373124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:08.373444 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:08.373689 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:08.373690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:08.373690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:08.373830 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:08.374155 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:08.374161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:08.374365 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 17:54:08.374494 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 17:54:08.374666 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 17:54:08.375705 13611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44813
I20260928 17:54:08.375831 13611 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.376019 13611 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:08.376785 13611 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44813 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 17:54:08.376861 13611 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.376920 13611 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:08.377287 13611 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.377363 13611 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
W20260928 17:54:08.377651 13611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:54:08.381987 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:08.364717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.0.98.62:44813)
0928 17:54:08.364853 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:08.364856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:08.368153 (+ 3297us) server_negotiation.cc:407] Connection header received
0928 17:54:08.368181 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:08.368182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:08.368196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:08.368275 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:08.375518 (+ 7243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.376301 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.377450 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:08.377770 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:08.377844 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:08.379054 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:08.379056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:08.379057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:08.379065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:08.380748 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.381132 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.381133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.381134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.381225 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:08.381469 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:08.381470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:08.381470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:08.381606 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:08.381764 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:08.381802 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 17:54:08.381887 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
Sep 28 17:54:08 dist-test-slave-zqlt krb5kdc[12149](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:08.965058 13343 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:08.965240 13343 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:08.965724 13145 ts_manager.cc:194] Registered new tserver with Master: 31251564c0864c8586cdf54af5029910 (127.0.98.1:41577)
I20260928 17:54:08.966763 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39549
I20260928 17:54:09.187325 13477 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:09.187501 13477 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:09.187832 13145 ts_manager.cc:194] Registered new tserver with Master: de33a051644f4eb6a6ba77c54864bc67 (127.0.98.2:45895)
I20260928 17:54:09.188433 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54625
Sep 28 17:54:09 dist-test-slave-zqlt krb5kdc[12149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 17:54:09.377859 13611 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:09.378023 13611 heartbeater.cc:511] Master 127.0.98.62:44813 requested a full tablet report, sending...
I20260928 17:54:09.378407 13145 ts_manager.cc:194] Registered new tserver with Master: 590157b62d814b84b030d205435d2284 (127.0.98.3:37693)
I20260928 17:54:09.378945 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36163
I20260928 17:54:09.384955 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:09.394871 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.386114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.98.62:44813)
0928 17:54:09.386479 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.386481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.386606 (+ 125us) server_negotiation.cc:407] Connection header received
0928 17:54:09.386722 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.386723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.386736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.386785 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:09.387623 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.388460 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.389549 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.389873 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.389918 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.391579 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:09.391581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:09.391581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:09.391589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:09.393509 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:09.393947 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:09.393949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:09.393949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:09.394009 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:09.394223 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:09.394224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:09.394225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:09.394353 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:09.394431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.394675 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.394761 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":68,"threads_started":1}
I20260928 17:54:09.397773 13145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:54:09.400549 13145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:54:09.411314 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.404203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41577 (local address 127.0.0.1:34850)
0928 17:54:09.405268 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.405275 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:09.405357 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:09.405608 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:09.405611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:09.405613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:09.405677 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.405679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.406948 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.406949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.409992 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.409994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.410990 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.410991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.411070 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.411083 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:54:09.411158 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:54:09.411208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":88,"threads_started":1}
I20260928 17:54:09.411759 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.404200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.0.98.1:41577)
0928 17:54:09.404862 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.404867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.405332 (+ 465us) server_negotiation.cc:407] Connection header received
0928 17:54:09.405436 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.405439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.405453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.405503 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:09.405749 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.406858 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.410074 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.410815 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.410863 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.410932 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.411620 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.411665 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":63,"threads_started":1}
I20260928 17:54:09.412695 13265 tablet_service.cc:1467] Processing CreateTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d (DEFAULT_TABLE table=test-table [id=1c2c504c53d844ab937d6fc827b1cfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:09.413048 13265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e54655f7e847be9ce455ddfeb8be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:09.419250 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.403701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37693 (local address 127.0.0.1:56784)
0928 17:54:09.404956 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.404995 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:09.405062 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:09.408623 (+ 3561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:09.408624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:09.408626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:09.408667 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.408669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.416787 (+ 8118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.416788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.417976 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.417979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.419002 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.419003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.419069 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.419073 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:09.419134 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:54:09.419174 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"thread_start_us":75,"threads_started":1}
I20260928 17:54:09.419695 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.404508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.98.3:37693)
0928 17:54:09.408401 (+ 3893us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.408406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.408417 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:54:09.408453 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.408455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.408471 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.408544 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:09.408745 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.409861 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.418076 (+ 8215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.418828 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.418893 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.418970 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.419578 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.419621 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3857,"thread_start_us":61,"threads_started":1}
I20260928 17:54:09.420414 13529 tablet_service.cc:1467] Processing CreateTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d (DEFAULT_TABLE table=test-table [id=1c2c504c53d844ab937d6fc827b1cfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:09.420715 13529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e54655f7e847be9ce455ddfeb8be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:09.422477 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.403912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45895 (local address 127.0.0.1:50920)
0928 17:54:09.405169 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.405177 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:09.405208 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:09.412430 (+ 7222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:09.412431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:09.412433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:09.412472 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.412473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.420184 (+ 7711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.420185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.421300 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.421304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.422264 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.422265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.422334 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.422338 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:09.422369 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 17:54:09.422405 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":51,"threads_started":1}
I20260928 17:54:09.422884 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.404655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.2:45895)
0928 17:54:09.412182 (+ 7527us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.412187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.412198 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:54:09.412236 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.412239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.412255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.412345 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:09.412548 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.413604 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.421412 (+ 7808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.422152 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.422638 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.422725 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.422766 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.422807 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7492,"thread_start_us":52,"threads_started":1}
I20260928 17:54:09.423458 13411 tablet_service.cc:1467] Processing CreateTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d (DEFAULT_TABLE table=test-table [id=1c2c504c53d844ab937d6fc827b1cfe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:09.423816 13411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e54655f7e847be9ce455ddfeb8be1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:09.424465 13644 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Bootstrap starting.
I20260928 17:54:09.425143 13644 tablet_bootstrap.cc:654] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:09.425971 13644 log.cc:824] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:09.427541 13645 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Bootstrap starting.
I20260928 17:54:09.428279 13645 tablet_bootstrap.cc:654] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:09.428634 13645 log.cc:824] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:09.428973 13647 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Bootstrap starting.
I20260928 17:54:09.429606 13647 tablet_bootstrap.cc:654] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:09.429955 13647 log.cc:824] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:09.430776 13644 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: No bootstrap required, opened a new log
I20260928 17:54:09.430897 13644 ts_tablet_manager.cc:1397] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 17:54:09.434068 13644 raft_consensus.cc:348] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.434370 13644 raft_consensus.cc:374] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:09.434465 13644 raft_consensus.cc:729] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31251564c0864c8586cdf54af5029910, State: Initialized, Role: FOLLOWER
I20260928 17:54:09.434676 13644 consensus_queue.cc:260] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.435067 13644 ts_tablet_manager.cc:1428] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 17:54:09.435236 13343 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
I20260928 17:54:09.436414 13645 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: No bootstrap required, opened a new log
I20260928 17:54:09.436539 13645 ts_tablet_manager.cc:1397] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 17:54:09.439309 13645 raft_consensus.cc:348] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.439527 13645 raft_consensus.cc:374] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:09.439612 13645 raft_consensus.cc:729] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590157b62d814b84b030d205435d2284, State: Initialized, Role: FOLLOWER
I20260928 17:54:09.439831 13645 consensus_queue.cc:260] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.440201 13645 ts_tablet_manager.cc:1428] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:54:09.440361 13611 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
I20260928 17:54:09.441728 13647 tablet_bootstrap.cc:492] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: No bootstrap required, opened a new log
I20260928 17:54:09.441872 13647 ts_tablet_manager.cc:1397] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 17:54:09.444656 13647 raft_consensus.cc:348] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.444907 13647 raft_consensus.cc:374] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:09.444991 13647 raft_consensus.cc:729] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de33a051644f4eb6a6ba77c54864bc67, State: Initialized, Role: FOLLOWER
I20260928 17:54:09.445191 13647 consensus_queue.cc:260] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.445555 13647 ts_tablet_manager.cc:1428] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:54:09.445709 13477 heartbeater.cc:503] Master 127.0.98.62:44813 was elected leader, sending a full tablet report...
W20260928 17:54:09.446895 13344 tablet.cc:2369] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:54:09.462002 13650 raft_consensus.cc:482] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:09.462320 13650 raft_consensus.cc:504] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.463860 13650 leader_election.cc:288] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 590157b62d814b84b030d205435d2284 (127.0.98.3:37693), de33a051644f4eb6a6ba77c54864bc67 (127.0.98.2:45895)
I20260928 17:54:09.470394 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.463381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44497 (local address 127.0.98.3:37693)
0928 17:54:09.463432 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.463435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.464152 (+ 717us) server_negotiation.cc:407] Connection header received
0928 17:54:09.464269 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.464271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.464285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.464316 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:09.464583 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.465358 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.469319 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.469954 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.470003 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.470018 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.470258 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.470311 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:54:09.470912 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.463168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37693 (local address 127.0.98.1:44497)
0928 17:54:09.464042 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.464054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:09.464187 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:09.464389 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:09.464390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:09.464392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:09.464492 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.464494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.465468 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.465468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.466687 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.466690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.470079 (+ 3389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.470080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.470173 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.470177 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:09.470739 (+ 562us) client_negotiation.cc:240] Negotiation successful
0928 17:54:09.470806 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":102,"threads_started":1}
I20260928 17:54:09.471385 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e54655f7e847be9ce455ddfeb8be1d" candidate_uuid: "31251564c0864c8586cdf54af5029910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590157b62d814b84b030d205435d2284" is_pre_election: true
I20260928 17:54:09.471565 13565 raft_consensus.cc:2454] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31251564c0864c8586cdf54af5029910 in term 0.
I20260928 17:54:09.471925 13231 leader_election.cc:302] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31251564c0864c8586cdf54af5029910, 590157b62d814b84b030d205435d2284; no voters:
I20260928 17:54:09.472164 13650 raft_consensus.cc:2781] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:09.472224 13650 raft_consensus.cc:482] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:09.472249 13650 raft_consensus.cc:3037] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:09.473198 13650 raft_consensus.cc:504] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.473333 13650 leader_election.cc:288] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [CANDIDATE]: Term 1 election: Requested vote from peers 590157b62d814b84b030d205435d2284 (127.0.98.3:37693), de33a051644f4eb6a6ba77c54864bc67 (127.0.98.2:45895)
I20260928 17:54:09.473623 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e54655f7e847be9ce455ddfeb8be1d" candidate_uuid: "31251564c0864c8586cdf54af5029910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590157b62d814b84b030d205435d2284"
I20260928 17:54:09.473719 13565 raft_consensus.cc:3037] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:09.474608 13565 raft_consensus.cc:2454] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31251564c0864c8586cdf54af5029910 in term 1.
I20260928 17:54:09.475327 13231 leader_election.cc:302] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31251564c0864c8586cdf54af5029910, 590157b62d814b84b030d205435d2284; no voters:
I20260928 17:54:09.475450 13650 raft_consensus.cc:2781] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:09.475628 13650 raft_consensus.cc:686] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [term 1 LEADER]: Becoming Leader. State: Replica: 31251564c0864c8586cdf54af5029910, State: Running, Role: LEADER
I20260928 17:54:09.475775 13650 consensus_queue.cc:237] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } }
I20260928 17:54:09.476701 13145 catalog_manager.cc:5495] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31251564c0864c8586cdf54af5029910 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "31251564c0864c8586cdf54af5029910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31251564c0864c8586cdf54af5029910" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41577 } health_report { overall_health: HEALTHY } } }
I20260928 17:54:09.478639 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.463793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40169 (local address 127.0.98.2:45895)
0928 17:54:09.463838 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.463840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.466888 (+ 3048us) server_negotiation.cc:407] Connection header received
0928 17:54:09.466991 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.466992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.467006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.467035 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:09.467225 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.467936 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.477263 (+ 9327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.477911 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.477974 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.477986 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.478508 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.478567 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:54:09.479046 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.463601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45895 (local address 127.0.98.1:40169)
0928 17:54:09.466826 (+ 3225us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.466838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:09.466916 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:09.467097 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:09.467098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:09.467099 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:09.467159 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.467161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.468037 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.468038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.469199 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:09.469202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.478329 (+ 9127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.478330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:09.478417 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.478421 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:09.478891 (+ 470us) client_negotiation.cc:240] Negotiation successful
0928 17:54:09.478947 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3205,"thread_start_us":75,"threads_started":1}
I20260928 17:54:09.479498 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e54655f7e847be9ce455ddfeb8be1d" candidate_uuid: "31251564c0864c8586cdf54af5029910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de33a051644f4eb6a6ba77c54864bc67" is_pre_election: true
I20260928 17:54:09.479652 13431 raft_consensus.cc:2454] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31251564c0864c8586cdf54af5029910 in term 0.
I20260928 17:54:09.479805 13430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e54655f7e847be9ce455ddfeb8be1d" candidate_uuid: "31251564c0864c8586cdf54af5029910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de33a051644f4eb6a6ba77c54864bc67"
I20260928 17:54:09.479875 13430 raft_consensus.cc:3037] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:09.480875 13430 raft_consensus.cc:2454] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31251564c0864c8586cdf54af5029910 in term 1.
I20260928 17:54:09.499178 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.495774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.98.1:41577)
0928 17:54:09.495821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:09.495823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:09.495832 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:09.495873 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:09.495874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:09.495899 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:09.495981 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:09.496209 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.496933 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.498032 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:09.498316 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:09.498387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:09.498587 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:09.498723 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:09.498835 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:09.499076 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 17:54:09.499098 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:54:09.501546 13430 raft_consensus.cc:1261] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 1 FOLLOWER]: Refusing update from remote peer 31251564c0864c8586cdf54af5029910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:09.502034 13655 consensus_queue.cc:1035] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de33a051644f4eb6a6ba77c54864bc67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:09.504590 13565 raft_consensus.cc:1261] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 1 FOLLOWER]: Refusing update from remote peer 31251564c0864c8586cdf54af5029910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:09.504966 13655 consensus_queue.cc:1035] T 57e54655f7e847be9ce455ddfeb8be1d P 31251564c0864c8586cdf54af5029910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "590157b62d814b84b030d205435d2284" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:09.509408 13658 mvcc.cc:205] Tried to move back new op lower bound from 7334371530755260416 to 7334371530653773824. Current Snapshot: MvccSnapshot[applied={T|T < 7334371530755260416}]
I20260928 17:54:09.509687 13661 mvcc.cc:205] Tried to move back new op lower bound from 7334371530755260416 to 7334371530653773824. Current Snapshot: MvccSnapshot[applied={T|T < 7334371530755260416}]
I20260928 17:54:09.512327 13659 mvcc.cc:205] Tried to move back new op lower bound from 7334371530755260416 to 7334371530653773824. Current Snapshot: MvccSnapshot[applied={T|T < 7334371530755260416}]
I20260928 17:54:09.515477 13145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:09.515581 13145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:09.516505 13145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20165934d1b648acbfd8794c2d22f971: Sending DeleteTablet for 3 replicas of tablet 57e54655f7e847be9ce455ddfeb8be1d
I20260928 17:54:09.517095 13529 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56784
I20260928 17:54:09.517364 13411 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50920
I20260928 17:54:09.517580 13265 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e54655f7e847be9ce455ddfeb8be1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34850
I20260928 17:54:09.518374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13183
I20260928 17:54:09.520205 13667 tablet_replica.cc:330] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: stopping tablet replica
I20260928 17:54:09.520359 13668 tablet_replica.cc:330] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: stopping tablet replica
I20260928 17:54:09.520594 13668 raft_consensus.cc:2229] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:09.520771 13667 raft_consensus.cc:2229] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:09.520934 13667 raft_consensus.cc:2258] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:09.521063 13668 raft_consensus.cc:2258] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:09.521718 13667 ts_tablet_manager.cc:1905] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:09.528805 13668 ts_tablet_manager.cc:1905] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:09.531087 13667 ts_tablet_manager.cc:1918] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:09.531342 13668 ts_tablet_manager.cc:1918] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:09.531497 13667 log.cc:1196] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/57e54655f7e847be9ce455ddfeb8be1d
I20260928 17:54:09.531921 13667 ts_tablet_manager.cc:1939] T 57e54655f7e847be9ce455ddfeb8be1d P 590157b62d814b84b030d205435d2284: Deleting consensus metadata
I20260928 17:54:09.532073 13668 log.cc:1196] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/57e54655f7e847be9ce455ddfeb8be1d
I20260928 17:54:09.532445 13668 ts_tablet_manager.cc:1939] T 57e54655f7e847be9ce455ddfeb8be1d P de33a051644f4eb6a6ba77c54864bc67: Deleting consensus metadata
I20260928 17:54:09.533038 13130 catalog_manager.cc:4841] TS 590157b62d814b84b030d205435d2284 (127.0.98.3:37693): tablet 57e54655f7e847be9ce455ddfeb8be1d (table test-table [id=1c2c504c53d844ab937d6fc827b1cfe5]) successfully deleted
I20260928 17:54:09.533608 13130 catalog_manager.cc:4841] TS de33a051644f4eb6a6ba77c54864bc67 (127.0.98.2:45895): tablet 57e54655f7e847be9ce455ddfeb8be1d (table test-table [id=1c2c504c53d844ab937d6fc827b1cfe5]) successfully deleted
W20260928 17:54:09.534204 13129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41577 (error 108)
I20260928 17:54:09.534883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13347
I20260928 17:54:09.542667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13480
I20260928 17:54:09.545074 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:09.544919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41577 (local address 127.0.0.1:34878)
0928 17:54:09.544980 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:09.545030 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 17:54:09.545645 13129 catalog_manager.cc:4584] TS 31251564c0864c8586cdf54af5029910 (127.0.98.1:41577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57e54655f7e847be9ce455ddfeb8be1d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41577: connect: Connection refused (error 111)
I20260928 17:54:09.550100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13112
I20260928 17:54:09.556686 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 17:54:09.901279 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 17:54:09.901376 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 17:54:10.246424 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 17:54:10.246 UTC [12179] LOG: received smart shutdown request
2026-09-28 17:54:10.250 UTC [12179] LOG: background worker "logical replication launcher" (PID 12188) exited with exit code 1
2026-09-28 17:54:10.250 UTC [12183] LOG: shutting down
2026-09-28 17:54:10.426 UTC [12179] LOG: database system is shut down
2026-09-28T17:54:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48839 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:54:10 dist-test-slave-zqlt krb5kdc[13679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:54:10 dist-test-slave-zqlt krb5kdc[13679](info): set up 2 sockets
Sep 28 17:54:10 dist-test-slave-zqlt krb5kdc[13679](info): commencing operation
krb5kdc: starting...
W20260928 17:54:11.572034 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:54:11.687386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:11.771190 13692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:11.771420 13692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:11.771443 13692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:11.772679 13692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:11.772720 13692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:11.772734 13692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:11.772747 13692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:11.772759 13692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:11.774217 13692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:11.774415 13692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:11.774602 13692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:11.779376 13697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:11.779342 13700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:11.779462 13692 server_base.cc:1034] running on GCE node
W20260928 17:54:11.779342 13698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:11.779872 13692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:11.780226 13692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:11.781389 13692 hybrid_clock.cc:648] HybridClock initialized: now 1790618051781375 us; error 46 us; skew 500 ppm
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:11.786695 13692 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:11.787132 13692 webserver.cc:460] Webserver started at http://127.0.98.62:38633/ using document root <none> and password file <none>
I20260928 17:54:11.787370 13692 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:11.787438 13692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:11.787590 13692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:11.788681 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "c2c6bb96955945cd8ff681f0e7537dd8"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "69547e175154031dd3ad06cc6c39bc6a"
server_key_iv: "de7811ee72a5fe11ba57db82ea3c94f1"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.789048 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2c6bb96955945cd8ff681f0e7537dd8"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "69547e175154031dd3ad06cc6c39bc6a"
server_key_iv: "de7811ee72a5fe11ba57db82ea3c94f1"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.790350 13692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:11.791052 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:11.791246 13692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:11.791352 13692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "c2c6bb96955945cd8ff681f0e7537dd8"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "69547e175154031dd3ad06cc6c39bc6a"
server_key_iv: "de7811ee72a5fe11ba57db82ea3c94f1"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.791424 13692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:11.810411 13692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:11.812266 13692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:11.812443 13692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:11.816365 13692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40969
I20260928 17:54:11.816403 13759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40969 every 8 connection(s)
I20260928 17:54:11.816761 13692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:11.817608 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:11.819206 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:54:11.822335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13692
I20260928 17:54:11.822444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:54:11.822623 392 external_mini_cluster.cc:1428] Setting key 437e543d7b7e2937f9872ce646139640
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 17:54:11.829816 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.820061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.98.62:40969)
0928 17:54:11.820262 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:11.820271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:11.820377 (+ 106us) server_negotiation.cc:407] Connection header received
0928 17:54:11.820932 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:11.820934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:11.820959 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:11.821072 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:11.821646 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.822702 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.823593 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.824024 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.824169 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.826539 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:11.826546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:11.826546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:11.826554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:11.828193 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.828593 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.828596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.828596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.828677 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.828898 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.828900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.828900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.829121 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:11.829213 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:11.829393 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 17:54:11.829566 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260928 17:54:11.829814 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.819908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:57238)
0928 17:54:11.820187 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.820261 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:11.820768 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:11.821088 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:11.821090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:11.821373 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:11.821520 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.821525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.822807 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.822809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.823497 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.823499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.824146 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.824147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.824214 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.824777 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:11.824785 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:11.826435 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:11.828287 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.828293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.828296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.828490 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.828745 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.828746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.828746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.828833 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.829191 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:11.829195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:11.829259 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 17:54:11.829371 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 17:54:11.829630 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":129,"thread_start_us":70,"threads_started":1}
I20260928 17:54:11.830744 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:54:11.831111 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.830987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49462)
0928 17:54:11.831035 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.831078 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:54:11.831252 13709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
I20260928 17:54:11.831607 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.831513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49476)
0928 17:54:11.831563 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.831586 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 17:54:11.831717 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 17:54:11.846457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40533
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:11.880944 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } attempt: 1
I20260928 17:54:11.881505 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.881346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49486)
0928 17:54:11.881409 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.881461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 17:54:11.881932 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.881809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49502)
0928 17:54:11.881863 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.881900 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 17:54:11.882046 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
W20260928 17:54:11.927919 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:11.928193 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:11.928220 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:11.929380 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:11.929416 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:11.929430 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:11.929445 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:11.929456 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:11.930960 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:11.931171 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:11.931350 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:11.936331 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:11.936311 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:11.936412 13767 server_base.cc:1034] running on GCE node
W20260928 17:54:11.936368 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:11.936801 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:11.937155 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:11.938325 13767 hybrid_clock.cc:648] HybridClock initialized: now 1790618051938315 us; error 44 us; skew 500 ppm
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:11.940826 13767 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 17:54:11.941267 13767 webserver.cc:460] Webserver started at http://127.0.98.61:42641/ using document root <none> and password file <none>
I20260928 17:54:11.941493 13767 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:11.941560 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:11.941720 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:11.942788 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data/instance:
uuid: "724f2d86ad6a4f3cb94946bb32d38d5e"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "04e4bdd9592af9da7b5dd31f54c3ede8"
server_key_iv: "d7fb3cef186df7fe38ab6862467a5fa4"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.943150 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal/instance:
uuid: "724f2d86ad6a4f3cb94946bb32d38d5e"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "04e4bdd9592af9da7b5dd31f54c3ede8"
server_key_iv: "d7fb3cef186df7fe38ab6862467a5fa4"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.944557 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:54:11.945344 13782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:11.945514 13767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:54:11.945622 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "724f2d86ad6a4f3cb94946bb32d38d5e"
format_stamp: "Formatted at 2026-09-28 17:54:11 on dist-test-slave-zqlt"
server_key: "04e4bdd9592af9da7b5dd31f54c3ede8"
server_key_iv: "d7fb3cef186df7fe38ab6862467a5fa4"
server_key_version: "encryptionkey@0"
I20260928 17:54:11.945721 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:11.950186 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } attempt: 2
I20260928 17:54:11.950609 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.950475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49514)
0928 17:54:11.950535 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.950568 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:54:11.951004 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.950890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49530)
0928 17:54:11.950942 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.950979 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:54:11.951143 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40533: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40533: connect: Connection refused (error 111)
I20260928 17:54:11.953434 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:11.955235 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:11.955380 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:11.959546 13767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40533
I20260928 17:54:11.959606 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40533 every 8 connection(s)
I20260928 17:54:11.959967 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
I20260928 17:54:11.960932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13767
I20260928 17:54:11.960983 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:11.961059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-1/wal/instance
I20260928 17:54:11.961300 392 external_mini_cluster.cc:1428] Setting key 2ece97f37300d3f05177f9357ee9c7c2
I20260928 17:54:11.962932 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:11.976130 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.963880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.98.62:40969)
0928 17:54:11.963951 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:11.963953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:11.964217 (+ 264us) server_negotiation.cc:407] Connection header received
0928 17:54:11.964751 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:11.964753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:11.964770 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:11.964856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:11.965551 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.966389 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.967406 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.967754 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.967837 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.972652 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:11.972654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:11.972654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:11.972662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:11.974396 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.974863 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.974864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.974865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.974930 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.975234 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.975236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.975236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.975406 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:11.975484 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:11.975859 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 17:54:11.975959 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:54:11.976284 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.963826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:57248)
0928 17:54:11.964052 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.964121 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:11.964589 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:11.964956 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:11.964959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:11.965227 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:11.965442 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.965447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.966495 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.966496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.967308 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.967310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.967853 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.967854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.967963 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.968668 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:11.968676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:11.972548 (+ 3872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:11.974496 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.974504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.974506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.974782 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.975001 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.975002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.975002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.975147 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.975500 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:11.975507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:11.975666 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 17:54:11.975822 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 17:54:11.976001 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260928 17:54:11.976897 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:54:11 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:54:11.984259 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.977182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49534)
0928 17:54:11.977230 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.977241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:11.977284 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:11.977651 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:11.977652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:11.977786 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:11.977835 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.977836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.978731 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.978732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.979218 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:11.979219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.979709 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.979710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:11.979775 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.980217 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:11.980222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:11.981718 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:11.983310 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.983311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.983311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.983461 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.983658 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:11.983659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:11.983659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:11.983725 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:11.983974 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:11.983974 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:11.984009 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 17:54:11.984058 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:54:11.984198 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:54:11.984441 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.977340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.98.61:40533)
0928 17:54:11.977481 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:11.977486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:11.977504 (+ 18us) server_negotiation.cc:407] Connection header received
0928 17:54:11.977531 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:11.977533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:11.977543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:11.977625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:11.977917 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.978635 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.979310 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:11.979620 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:11.979710 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:11.981794 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:11.981801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:11.981801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:11.981808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:11.983243 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.983527 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.983529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.983529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.983579 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:11.983782 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:11.983783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:11.983783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:11.983915 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:11.983981 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:11.984081 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:54:11.984299 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260928 17:54:11.985073 13835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:54:11.985428 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.985313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:44994)
0928 17:54:11.985358 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.985401 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 17:54:11.985548 13786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
I20260928 17:54:11.985971 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:11.985883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45006)
0928 17:54:11.985928 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:11.985952 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:54:11.986073 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 17:54:11.986811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33021
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:12.035328 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } attempt: 1
I20260928 17:54:12.035907 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.035762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45008)
0928 17:54:12.035826 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.035871 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:54:12.036300 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.036206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45014)
0928 17:54:12.036252 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.036279 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 17:54:12.036398 13835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
I20260928 17:54:12.042367 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } attempt: 3
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:54:12.049507 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.042741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49538)
0928 17:54:12.042784 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.042796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.042853 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.043017 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.043018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.043177 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.043275 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.043277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.043995 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.043997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.044570 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.044572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.044999 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.045000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.045088 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.045588 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.045593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.046776 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.048409 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.048410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.048411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.048599 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.048828 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.048828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.048829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.048895 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.049157 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.049158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.049236 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.049300 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.049409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 17:54:12.049507 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.042783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.98.61:40533)
0928 17:54:12.042845 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.042847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.042858 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:54:12.042918 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.042919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.042931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.043000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.043365 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.043889 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.044674 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.044891 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.045000 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.046868 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.046869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.046870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.046878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.048314 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.048685 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.048687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.048687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.048753 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.048959 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.048960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.048961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.049079 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.049153 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.049335 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.049413 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:54:12.049911 13760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:54:12.050279 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.050165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45030)
0928 17:54:12.050221 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.050249 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:54:12.050683 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.050611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45036)
0928 17:54:12.050647 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.050669 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 17:54:12.050796 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
W20260928 17:54:12.069633 13842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:12.069846 13842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:12.069868 13842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:12.071053 13842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:12.071099 13842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:12.071117 13842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:12.071131 13842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:12.071183 13842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:12.072819 13842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40969,127.0.98.61:40533,127.0.98.60:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:12.073078 13842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:12.073319 13842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:12.078157 13848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:12.078167 13847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:12.078177 13850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:12.078246 13842 server_base.cc:1034] running on GCE node
I20260928 17:54:12.078583 13842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:12.078881 13842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:12.080029 13842 hybrid_clock.cc:648] HybridClock initialized: now 1790618052080017 us; error 27 us; skew 500 ppm
I20260928 17:54:12.081921 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } attempt: 1
I20260928 17:54:12.082381 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.082245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45038)
0928 17:54:12.082311 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.082346 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:12.082765 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.082635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45052)
0928 17:54:12.082689 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.082720 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 17:54:12.082872 13760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33021: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33021: connect: Connection refused (error 111)
I20260928 17:54:12.082919 13842 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 17:54:12.083341 13842 webserver.cc:460] Webserver started at http://127.0.98.60:35483/ using document root <none> and password file <none>
I20260928 17:54:12.083621 13842 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:12.083668 13842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:12.083808 13842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:12.084950 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data/instance:
uuid: "54f0b785d54d4a2cb76043a2840cacea"
format_stamp: "Formatted at 2026-09-28 17:54:12 on dist-test-slave-zqlt"
server_key: "56a3065ef46e8cc270dca49215f1895b"
server_key_iv: "2b9b7bbd45726b38dd0d2c911ff14d5f"
server_key_version: "encryptionkey@0"
I20260928 17:54:12.085338 13842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal/instance:
uuid: "54f0b785d54d4a2cb76043a2840cacea"
format_stamp: "Formatted at 2026-09-28 17:54:12 on dist-test-slave-zqlt"
server_key: "56a3065ef46e8cc270dca49215f1895b"
server_key_iv: "2b9b7bbd45726b38dd0d2c911ff14d5f"
server_key_version: "encryptionkey@0"
I20260928 17:54:12.086683 13842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:54:12.087385 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:12.087556 13842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:12.087659 13842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "54f0b785d54d4a2cb76043a2840cacea"
format_stamp: "Formatted at 2026-09-28 17:54:12 on dist-test-slave-zqlt"
server_key: "56a3065ef46e8cc270dca49215f1895b"
server_key_iv: "2b9b7bbd45726b38dd0d2c911ff14d5f"
server_key_version: "encryptionkey@0"
I20260928 17:54:12.087713 13842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:12.097436 13842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:12.099669 13842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:12.099781 13842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:12.103760 13842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33021
I20260928 17:54:12.103859 13909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33021 every 8 connection(s)
I20260928 17:54:12.104239 13842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
I20260928 17:54:12.104532 13835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } attempt: 2
I20260928 17:54:12.105376 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:12.107550 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:54:12.111464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13842
I20260928 17:54:12.111595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790617898245922-392-0/minicluster-data/master-2/wal/instance
I20260928 17:54:12.111850 392 external_mini_cluster.cc:1428] Setting key 7c892c74de44a6e85af68eb83fdba371
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:12.116061 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.104817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45060)
0928 17:54:12.104875 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.104887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.104936 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.106081 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.106082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.106239 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.106284 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.106286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.107884 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.107885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.108698 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.108711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.109481 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.109482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.109543 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.110022 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.110027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.111486 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.114739 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.114740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.114740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.114925 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.115193 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.115194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.115265 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.115717 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.115717 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.115764 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.115837 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.115950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 17:54:12.116331 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.105003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.0.98.60:33021)
0928 17:54:12.105198 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.105207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.105240 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:54:12.105850 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.105860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.105922 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.106100 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.106393 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.107781 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.108791 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.109408 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.109486 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.112423 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.112432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.112432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.112439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.114630 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115020 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.115024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.115024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.115116 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115352 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.115353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.115354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.115661 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.115734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.115915 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.116055 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 17:54:12.116814 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.108234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40969 (local address 127.0.0.1:57260)
0928 17:54:12.108418 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.108458 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.108531 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.108692 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.108694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.109053 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.109130 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.109132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.109914 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.109915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.110643 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.110646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.110998 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.110999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.111079 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.111647 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.111664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.113377 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.115306 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.115315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.115572 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.115783 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.115785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.115848 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.116503 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.116508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.116556 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.116640 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.116737 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260928 17:54:12.116886 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.108325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.0.98.62:40969)
0928 17:54:12.108393 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.108395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.108523 (+ 128us) server_negotiation.cc:407] Connection header received
0928 17:54:12.108598 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.108599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.108617 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.108675 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.109210 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.109820 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.110731 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.110913 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.110968 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.113479 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.113481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.113481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.113489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.115225 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115655 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.115657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.115657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.115728 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.115929 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.115931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.115931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.116401 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.116452 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.116696 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.116781 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:54:12.117590 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:54:12.118731 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: Bootstrap starting.
I20260928 17:54:12.119616 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:12.119946 13835 log.cc:824] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:12.120922 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: No bootstrap required, opened a new log
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:54:12.122959 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.112254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.98.60:33021)
0928 17:54:12.112393 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.112395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.112515 (+ 120us) server_negotiation.cc:407] Connection header received
0928 17:54:12.112578 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.112579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.112590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.112669 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.113689 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.114328 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.115379 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.115590 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.115672 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.119571 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.119573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.119574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.119581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.121468 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.121869 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.121870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.121871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.121931 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.122162 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.122163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.122164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.122276 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.122425 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.122778 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.122863 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260928 17:54:12.123425 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.123672 13835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:12.123736 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724f2d86ad6a4f3cb94946bb32d38d5e, State: Initialized, Role: FOLLOWER
I20260928 17:54:12.123903 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.124270 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:12.125204 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.117860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40533 (local address 127.0.0.1:49546)
0928 17:54:12.118076 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.118085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.118147 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.118271 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.118272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.118404 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.118449 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.118451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.119163 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.119164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.119731 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.119733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.120242 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.120243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.120322 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.120763 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.120768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.122195 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.124054 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.124056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.124056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.124289 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.124514 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.124515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.124515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.124575 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.124910 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.124911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.124951 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.125008 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.125100 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260928 17:54:12.125543 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.117929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.0.98.61:40533)
0928 17:54:12.117984 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.117987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.118133 (+ 146us) server_negotiation.cc:407] Connection header received
0928 17:54:12.118205 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.118206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.118216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.118259 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.118521 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.119082 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.119805 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.120131 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.120192 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.122296 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.122298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.122298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.122305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.123934 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.124378 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.124379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.124379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.124436 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.124649 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.124651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.124651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.124816 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.124881 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.125377 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.125453 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:54:12.125823 13910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } has no permanent_uuid. Determining permanent_uuid...
W20260928 17:54:12.126976 13928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:54:12.127107 13928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:12.126537 13767 master_runner.cc:186] Error getting master registration for 127.0.98.62:40969: OK, instance_id { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" instance_seqno: 1790618051815271 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:54:12.129560 13767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:54:12.132195 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.132388 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.133183 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.123634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.98.62:40969)
0928 17:54:12.123673 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.123675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.123683 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:54:12.123705 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.123706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.123715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.123761 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.124417 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.125067 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.126423 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.126728 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.126788 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.129768 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.129770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.129771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.129778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.131503 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.131906 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.131907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.131908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.131969 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.132244 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.132246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.132246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.132385 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.132565 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.132988 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.133078 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:54:12.133383 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.126106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45084)
0928 17:54:12.126149 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.126159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.126193 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.126386 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.126387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.126526 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.126556 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.126557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.127422 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.127423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.128280 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.128284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.128749 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.128750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.128798 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.129204 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.129209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.130566 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.132280 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.132281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.132281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.132476 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.132677 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.132678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.132678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.132732 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.133006 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.133006 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.133039 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.133085 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.133264 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 17:54:12.133750 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.126167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.0.98.60:33021)
0928 17:54:12.126199 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.126201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.126209 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:54:12.126243 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.126243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.126260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.126326 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.126627 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.127333 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.128368 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.128665 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.128929 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.130689 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.130691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.130691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.130698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.132200 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.132550 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.132551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.132551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.132601 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.132805 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.132806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.132806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.132934 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.133550 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.133598 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.133670 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:54:12.135216 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: Bootstrap starting.
I20260928 17:54:12.135948 13910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:12.136318 13910 log.cc:824] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:12.137231 13910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: No bootstrap required, opened a new log
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:54:12.139225 13910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.139392 13910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:12.139441 13910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f0b785d54d4a2cb76043a2840cacea, State: Initialized, Role: FOLLOWER
I20260928 17:54:12.139568 13910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.139890 13910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:12.140286 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.142637 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.133901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.0.98.61:40533)
0928 17:54:12.133940 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.133943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.133951 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:54:12.133974 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.133975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.133984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.134044 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.134631 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.135299 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.136279 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.136561 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.136615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.139375 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.139376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.139377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.139385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.141110 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.141493 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.141494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.141494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.141549 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.141738 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.141739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.141739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.141868 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.142027 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.142460 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.142543 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:54:12.140429 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.144205 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 17:54:12.144399 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 17:54:12.141927 13842 master_runner.cc:186] Error getting master registration for 127.0.98.62:40969: OK, instance_id { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" instance_seqno: 1790618051815271 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 17:54:12.144446 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:54:12.144492 13842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:54:12.154167 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.145712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.98.62:40969)
0928 17:54:12.145756 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.145758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.146061 (+ 303us) server_negotiation.cc:407] Connection header received
0928 17:54:12.146119 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.146120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.146132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.146201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.146912 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.147665 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.148799 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.149097 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.149176 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.150798 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.150799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.150800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.150808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.152930 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.153325 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.153326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.153326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.153393 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.153613 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.153614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.153614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.153753 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.153880 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.153984 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.154068 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:54:12.157992 13760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } attempt: 2
I20260928 17:54:12.161473 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.152725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.0.98.60:33021)
0928 17:54:12.152799 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.152801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.154199 (+ 1398us) server_negotiation.cc:407] Connection header received
0928 17:54:12.154299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.154300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.154311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.154343 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.154981 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.155706 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.156761 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.157066 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.157141 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.158586 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.158587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.158587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.158595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.160257 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.160654 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.160655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.160656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.160710 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.160899 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.160900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.160900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.161025 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.161097 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.161297 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.161375 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 17:54:12 dist-test-slave-zqlt krb5kdc[13679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:54:12.165748 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.152679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.0.98.61:40533)
0928 17:54:12.152769 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.152771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.156226 (+ 3455us) server_negotiation.cc:407] Connection header received
0928 17:54:12.156328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.156330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.156341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.156371 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.157050 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.158072 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.159338 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.159873 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.159914 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.161365 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.161366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.161367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.161374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.163855 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.164318 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.164319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.164320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.164379 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.164848 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.164849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.164849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.164993 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.165331 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.165550 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.165649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 17:54:12.167199 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.158291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33021 (local address 127.0.0.1:45108)
0928 17:54:12.158426 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:12.158437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:12.158476 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:12.158726 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:12.158727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:12.158893 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:12.158935 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.158937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.159857 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.159858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.160674 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:12.160677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.161253 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.161254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:12.161313 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.161936 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:12.161942 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:12.164250 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:12.166103 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.166105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.166105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.166315 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.166541 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:12.166542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:12.166542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:12.166617 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:12.166892 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:12.166893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:12.166937 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:54:12.166998 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:54:12.167094 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 17:54:12.167703 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:12.158569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.0.98.60:33021)
0928 17:54:12.158610 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:12.158612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:12.158618 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:54:12.158637 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:12.158638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:12.158646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:12.158690 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:12.159017 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.159768 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.160775 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:12.161138 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:12.161201 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:12.164344 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:12.164346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:12.164346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:12.164353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:12.165994 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.166392 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.166394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.166394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.166453 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:12.166685 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:12.166686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:12.166686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:12.166810 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:12.166866 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:12.167513 (+ 647us) server_negotiation.cc:299] Negotiation successful
0928 17:54:12.167592 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:54:12.169564 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: Bootstrap starting.
I20260928 17:54:12.170230 13760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:12.170529 13760 log.cc:824] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:12.171454 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: No bootstrap required, opened a new log
I20260928 17:54:12.173573 13760 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.173736 13760 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:12.173771 13760 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c6bb96955945cd8ff681f0e7537dd8, State: Initialized, Role: FOLLOWER
I20260928 17:54:12.173954 13760 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.174239 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.174289 13760 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:12.175760 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:54:12.176244 13963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:54:12.176331 13963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:54:12.179426 13692 master_runner.cc:263] No leader master found from master c2c6bb96955945cd8ff681f0e7537dd8
I20260928 17:54:12.179479 13692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 17:54:12.206089 13788 tablet.cc:2369] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:54:12.287825 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:12.287957 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.288229 13917 leader_election.cc:288] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2c6bb96955945cd8ff681f0e7537dd8 (127.0.98.62:40969), 54f0b785d54d4a2cb76043a2840cacea (127.0.98.60:33021)
I20260928 17:54:12.288538 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" is_pre_election: true
I20260928 17:54:12.288537 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f0b785d54d4a2cb76043a2840cacea" is_pre_election: true
I20260928 17:54:12.288697 13734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724f2d86ad6a4f3cb94946bb32d38d5e in term 0.
I20260928 17:54:12.288723 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724f2d86ad6a4f3cb94946bb32d38d5e in term 0.
I20260928 17:54:12.288921 13786 leader_election.cc:302] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f0b785d54d4a2cb76043a2840cacea, 724f2d86ad6a4f3cb94946bb32d38d5e; no voters:
I20260928 17:54:12.289104 13917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:12.289188 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:12.289217 13917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:12.290076 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.290215 13917 leader_election.cc:288] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [CANDIDATE]: Term 1 election: Requested vote from peers c2c6bb96955945cd8ff681f0e7537dd8 (127.0.98.62:40969), 54f0b785d54d4a2cb76043a2840cacea (127.0.98.60:33021)
I20260928 17:54:12.290418 13734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2c6bb96955945cd8ff681f0e7537dd8"
I20260928 17:54:12.290423 13884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54f0b785d54d4a2cb76043a2840cacea"
I20260928 17:54:12.290494 13734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:12.290494 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:12.291410 13734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724f2d86ad6a4f3cb94946bb32d38d5e in term 1.
I20260928 17:54:12.291406 13884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724f2d86ad6a4f3cb94946bb32d38d5e in term 1.
I20260928 17:54:12.291579 13786 leader_election.cc:302] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f0b785d54d4a2cb76043a2840cacea, 724f2d86ad6a4f3cb94946bb32d38d5e; no voters:
I20260928 17:54:12.291684 13917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:12.291918 13917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [term 1 LEADER]: Becoming Leader. State: Replica: 724f2d86ad6a4f3cb94946bb32d38d5e, State: Running, Role: LEADER
I20260928 17:54:12.292026 13917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } }
I20260928 17:54:12.292418 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724f2d86ad6a4f3cb94946bb32d38d5e. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.292557 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:12.292774 13967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:12.292989 13967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:12.294386 13884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [term 1 FOLLOWER]: Refusing update from remote peer 724f2d86ad6a4f3cb94946bb32d38d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:12.294375 13734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [term 1 FOLLOWER]: Refusing update from remote peer 724f2d86ad6a4f3cb94946bb32d38d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:12.294674 13965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:12.294757 13917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:12.295951 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724f2d86ad6a4f3cb94946bb32d38d5e. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.296066 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.296154 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724f2d86ad6a4f3cb94946bb32d38d5e. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.296258 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.297114 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.297207 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:12.297202 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.297269 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.297428 13968 mvcc.cc:205] Tried to move back new op lower bound from 7334371542195650560 to 7334371542188654592. Current Snapshot: MvccSnapshot[applied={T|T < 7334371542195650560}]
I20260928 17:54:12.297853 13970 mvcc.cc:205] Tried to move back new op lower bound from 7334371542195650560 to 7334371542188654592. Current Snapshot: MvccSnapshot[applied={T|T < 7334371542195650560}]
I20260928 17:54:12.298753 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.298828 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:12.298986 13967 catalog_manager.cc:1347] Generated new cluster ID: 6c070e9bb5aa4b6a8b6bddb091040448
I20260928 17:54:12.299029 13967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:12.299377 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2c6bb96955945cd8ff681f0e7537dd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40969 } } peers { permanent_uuid: "724f2d86ad6a4f3cb94946bb32d38d5e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40533 } } peers { permanent_uuid: "54f0b785d54d4a2cb76043a2840cacea" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33021 } } }
I20260928 17:54:12.299445 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:54:12.299705 13969 mvcc.cc:205] Tried to move back new op lower bound from 7334371542195650560 to 7334371542188654592. Current Snapshot: MvccSnapshot[applied={T|T < 7334371542195650560}]
I20260928 17:54:12.313354 13967 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:54:12.314132 13967 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:12.326176 13967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724f2d86ad6a4f3cb94946bb32d38d5e: Generated new TSK 0
I20260928 17:54:12.326475 13967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:54:13.144933 13942 catalog_manager.cc:1259] Loaded cluster ID: 6c070e9bb5aa4b6a8b6bddb091040448
I20260928 17:54:13.145012 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: loading cluster ID for follower catalog manager: success
I20260928 17:54:13.145949 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: acquiring CA information for follower catalog manager: success
I20260928 17:54:13.146246 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54f0b785d54d4a2cb76043a2840cacea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:54:13.176769 13963 catalog_manager.cc:1259] Loaded cluster ID: 6c070e9bb5aa4b6a8b6bddb091040448
I20260928 17:54:13.176847 13963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: loading cluster ID for follower catalog manager: success
I20260928 17:54:13.177630 13963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: acquiring CA information for follower catalog manager: success
I20260928 17:54:13.177907 13963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2c6bb96955945cd8ff681f0e7537dd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:54:13.384516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13692
I20260928 17:54:13.389940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13767
I20260928 17:54:13.394995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13842
2026-09-28T17:54:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2863 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:54:13 dist-test-slave-zqlt krb5kdc[13983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:54:13 dist-test-slave-zqlt krb5kdc[13983](info): set up 2 sockets
Sep 28 17:54:13 dist-test-slave-zqlt krb5kdc[13983](info): commencing operation
krb5kdc: starting...
W20260928 17:54:14.432260 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:54:14.563242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:14.646103 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:14.646324 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:14.646346 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:14.647562 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:14.647606 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:14.647621 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:14.647634 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:14.647646 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:14.649266 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:14.649552 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:14.649806 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:14.654917 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:14.654973 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:14.655128 13996 server_base.cc:1034] running on GCE node
W20260928 17:54:14.654925 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:14.655418 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:14.655727 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:14.656872 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790618054656857 us; error 45 us; skew 500 ppm
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[13983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:14.659525 13996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:14.659938 13996 webserver.cc:460] Webserver started at http://127.0.98.62:43139/ using document root <none> and password file <none>
I20260928 17:54:14.660205 13996 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:14.660254 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:14.660352 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:14.661368 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "2c05e1c7e0ea481fb0d392b538d6206e"
format_stamp: "Formatted at 2026-09-28 17:54:14 on dist-test-slave-zqlt"
server_key: "cea6c3d77afd43ff86ebcaa5d4cfc61d"
server_key_iv: "1bb83821b99cf63718da86ce7fbc7851"
server_key_version: "encryptionkey@0"
I20260928 17:54:14.661665 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c05e1c7e0ea481fb0d392b538d6206e"
format_stamp: "Formatted at 2026-09-28 17:54:14 on dist-test-slave-zqlt"
server_key: "cea6c3d77afd43ff86ebcaa5d4cfc61d"
server_key_iv: "1bb83821b99cf63718da86ce7fbc7851"
server_key_version: "encryptionkey@0"
I20260928 17:54:14.662932 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:54:14.663686 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:14.663887 13996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:14.663995 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "2c05e1c7e0ea481fb0d392b538d6206e"
format_stamp: "Formatted at 2026-09-28 17:54:14 on dist-test-slave-zqlt"
server_key: "cea6c3d77afd43ff86ebcaa5d4cfc61d"
server_key_iv: "1bb83821b99cf63718da86ce7fbc7851"
server_key_version: "encryptionkey@0"
I20260928 17:54:14.664072 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:14.690395 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:14.692575 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:14.692756 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:14.696756 13996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46863
I20260928 17:54:14.696838 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46863 every 8 connection(s)
I20260928 17:54:14.697194 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:14.697978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 13996
I20260928 17:54:14.698097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:54:14.698241 14064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:14.698333 392 external_mini_cluster.cc:1428] Setting key e48ce9fd50d769d5acc1e08ffee5ec37
I20260928 17:54:14.701850 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e: Bootstrap starting.
I20260928 17:54:14.702575 14064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:14.702888 14064 log.cc:824] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:14.703586 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e: No bootstrap required, opened a new log
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[13983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:14.705577 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260928 17:54:14.705752 14064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:14.705821 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c05e1c7e0ea481fb0d392b538d6206e, State: Initialized, Role: FOLLOWER
I20260928 17:54:14.705982 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260928 17:54:14.706073 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:14.706113 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:14.706157 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:14.706838 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260928 17:54:14.706959 14064 leader_election.cc:302] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c05e1c7e0ea481fb0d392b538d6206e; no voters:
I20260928 17:54:14.707175 14064 leader_election.cc:288] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:54:14.707211 14069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:14.707315 14069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [term 1 LEADER]: Becoming Leader. State: Replica: 2c05e1c7e0ea481fb0d392b538d6206e, State: Running, Role: LEADER
I20260928 17:54:14.707461 14069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260928 17:54:14.707468 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:14.708230 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c05e1c7e0ea481fb0d392b538d6206e. Latest consensus state: current_term: 1 leader_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260928 17:54:14.708395 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:14.708679 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:14.709385 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:14.709617 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c05e1c7e0ea481fb0d392b538d6206e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260928 17:54:14.709843 14067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:14.698847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.0.98.62:46863)
0928 17:54:14.699026 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:14.699034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:14.699060 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:54:14.699487 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:14.699494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:14.699544 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:14.699662 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:14.700495 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:14.701400 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:14.702465 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:14.702918 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:14.703040 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:14.705547 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:14.705552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:14.705552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:14.705571 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:14.707655 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:14.708134 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:14.708137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:14.708138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:14.708279 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:14.708537 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:14.708538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:14.708538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:14.708864 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:14.709031 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:14.709110 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:54:14.709295 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260928 17:54:14.709870 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:14.710891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:54:14.711591 14079 catalog_manager.cc:1347] Generated new cluster ID: ae27dfd96ead48cfbfa11bfe6779b7c4
I20260928 17:54:14.711639 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:14.724504 14079 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:54:14.725626 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:14.732847 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c05e1c7e0ea481fb0d392b538d6206e: Generated new TSK 0
I20260928 17:54:14.733069 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:54:14.767637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 13996
2026-09-28T17:54:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1371 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[14091](info): set up 2 sockets
Sep 28 17:54:14 dist-test-slave-zqlt krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260928 17:54:15.796257 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:54:15 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:54:15.914770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:15.994427 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:15.994683 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:15.994721 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:15.995899 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:15.995965 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:15.995990 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:15.996018 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:15.996049 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:15.997560 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:15.997833 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:15.998070 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.002985 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.003087 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.002959 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.003151 14104 server_base.cc:1034] running on GCE node
I20260928 17:54:16.003468 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.003762 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.004911 14104 hybrid_clock.cc:648] HybridClock initialized: now 1790618056004888 us; error 47 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.007381 14104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:16.007879 14104 webserver.cc:460] Webserver started at http://127.0.98.62:35549/ using document root <none> and password file <none>
I20260928 17:54:16.008188 14104 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.008261 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.008419 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:16.009490 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "fca03e5923ae4b3a843dc9c6fcd226aa"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "dca8b79b3b2c6d4fe698622cfd067383"
server_key_iv: "8430a7623d55f026856e6a657b5476e9"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.009858 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "fca03e5923ae4b3a843dc9c6fcd226aa"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "dca8b79b3b2c6d4fe698622cfd067383"
server_key_iv: "8430a7623d55f026856e6a657b5476e9"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.011219 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:54:16.011940 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.012156 14104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:16.012266 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "fca03e5923ae4b3a843dc9c6fcd226aa"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "dca8b79b3b2c6d4fe698622cfd067383"
server_key_iv: "8430a7623d55f026856e6a657b5476e9"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.012356 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.018802 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.020857 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.021029 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.025333 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45315
I20260928 17:54:16.025363 14171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45315 every 8 connection(s)
I20260928 17:54:16.025736 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:16.026741 14172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:16.029124 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap starting.
I20260928 17:54:16.029804 14172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:16.029783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14104
I20260928 17:54:16.029919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:54:16.030269 392 external_mini_cluster.cc:1428] Setting key f6829db111064765ccb24806d72c59a9
I20260928 17:54:16.030294 14172 log.cc:824] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:16.031272 14172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: No bootstrap required, opened a new log
I20260928 17:54:16.033960 14172 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.034171 14172 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:16.034207 14172 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Initialized, Role: FOLLOWER
I20260928 17:54:16.034338 14172 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.034417 14172 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:16.034447 14172 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:16.034494 14172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:16.035236 14172 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.035328 14172 leader_election.cc:302] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca03e5923ae4b3a843dc9c6fcd226aa; no voters:
I20260928 17:54:16.035597 14172 leader_election.cc:288] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:54:16.035621 14177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:16.035892 14172 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:16.036137 14177 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 LEADER]: Becoming Leader. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Running, Role: LEADER
I20260928 17:54:16.036273 14177 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.036718 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca03e5923ae4b3a843dc9c6fcd226aa. Latest consensus state: current_term: 1 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:16.036737 14179 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:16.037081 14179 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:16.036890 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:16.037431 14189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:16.037987 14189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:16.039757 14189 catalog_manager.cc:1347] Generated new cluster ID: 6b43db108f394267a73cee536ff88b6d
I20260928 17:54:16.039811 14189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:16.042191 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.031038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.98.62:45315)
0928 17:54:16.031226 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.031238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.031269 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:54:16.031806 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.031815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.031867 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.031993 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:16.032897 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.033812 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.034901 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.035352 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.035462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.038579 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:16.038583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:16.038583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:16.038595 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:16.040630 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.041042 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.041046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.041046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.041141 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.041351 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.041352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.041352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.041597 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:16.041756 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.041860 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.042017 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 17:54:16.045560 14189 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:54:16.046084 14189 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:16.053408 14189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Generated new TSK 0
I20260928 17:54:16.053633 14189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:54:16.084841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:16.182999 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:16.183228 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:16.183264 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:16.184507 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:16.184587 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:16.184634 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:16.186098 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:16.186362 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:16.186621 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.192301 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.192337 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.192310 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.192587 14198 server_base.cc:1034] running on GCE node
I20260928 17:54:16.192787 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.193073 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.194255 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790618056194236 us; error 35 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.196887 14198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:16.197271 14198 webserver.cc:460] Webserver started at http://127.0.98.1:36795/ using document root <none> and password file <none>
I20260928 17:54:16.197458 14198 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.197502 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.197589 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:16.198601 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "4369a573873441dca24fb6bbad585ed9"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "9b5b80ae5d787b56f55b15d8483789f4"
server_key_iv: "c5d0ddf4b248cccc6ce3ea6114b26595"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.198977 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4369a573873441dca24fb6bbad585ed9"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "9b5b80ae5d787b56f55b15d8483789f4"
server_key_iv: "c5d0ddf4b248cccc6ce3ea6114b26595"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.200395 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:54:16.201210 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.201429 14198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:16.201514 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "4369a573873441dca24fb6bbad585ed9"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "9b5b80ae5d787b56f55b15d8483789f4"
server_key_iv: "c5d0ddf4b248cccc6ce3ea6114b26595"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.201613 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.217671 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.220233 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.220356 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.220575 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:16.220911 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:16.220943 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.220965 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:16.220978 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.226712 14198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40817
I20260928 17:54:16.226795 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40817 every 8 connection(s)
I20260928 17:54:16.227131 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:16.230407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14198
I20260928 17:54:16.230528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:54:16.230729 392 external_mini_cluster.cc:1428] Setting key b171aa847752517cdf713ff2621da3de
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:16.237957 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.227866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48877 (local address 127.0.98.62:45315)
0928 17:54:16.227937 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.227940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.228217 (+ 277us) server_negotiation.cc:407] Connection header received
0928 17:54:16.228758 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.228759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.228770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.228831 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:16.229771 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.230416 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.231697 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.232032 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.232137 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.234741 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:16.234742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:16.234742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:16.234750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:16.236488 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.236926 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.236927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.236928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.236992 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.237303 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.237303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.237304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.237423 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:16.237492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.237777 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.237861 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:54:16.238173 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.227850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:48877)
0928 17:54:16.228036 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.228079 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.228635 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.228887 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.228890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.229455 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:16.229660 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.229665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.230529 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.230531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.231618 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.231621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.232168 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.232169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.232278 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.232953 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:16.232965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:16.234629 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:16.236584 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.236591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.236593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.236826 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.237108 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.237109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.237109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.237226 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.237495 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:16.237502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:16.237643 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.237754 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.237917 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260928 17:54:16.238652 14327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:16.238770 14327 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:16.238999 14327 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:16.239580 14136 ts_manager.cc:194] Registered new tserver with Master: 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:54:16.240291 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:54:16.256960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:16.339762 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:16.339991 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:16.340016 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:16.341360 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:16.341403 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:16.341431 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:16.342895 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:16.343096 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:16.343283 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.348404 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.348376 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.348376 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.348665 14332 server_base.cc:1034] running on GCE node
I20260928 17:54:16.348875 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.349129 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.350274 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790618056350266 us; error 31 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.352974 14332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:16.353462 14332 webserver.cc:460] Webserver started at http://127.0.98.2:36129/ using document root <none> and password file <none>
I20260928 17:54:16.353700 14332 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.353767 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.353917 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:16.355013 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "afc52d74796478b9be5397c1d730551c"
server_key_iv: "8a3fe4f4647c67d9587711705b7cde91"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.355386 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "afc52d74796478b9be5397c1d730551c"
server_key_iv: "8a3fe4f4647c67d9587711705b7cde91"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.356832 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:54:16.357643 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.357862 14332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:16.357968 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "afc52d74796478b9be5397c1d730551c"
server_key_iv: "8a3fe4f4647c67d9587711705b7cde91"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.358047 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.373890 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.375841 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.376010 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.376291 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:16.376614 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:16.376669 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.376704 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:16.376732 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.382695 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46235
I20260928 17:54:16.382750 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46235 every 8 connection(s)
I20260928 17:54:16.383095 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:16.391762 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.383851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35653 (local address 127.0.98.62:45315)
0928 17:54:16.383916 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.383918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.384138 (+ 220us) server_negotiation.cc:407] Connection header received
0928 17:54:16.384657 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.384658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.384668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.384727 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:16.385742 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.386242 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.387205 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.387378 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.387424 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.389468 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:16.389470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:16.389470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:16.389476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:16.390637 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.390981 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.390981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.390982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.391028 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.391276 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.391277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.391277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.391370 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:16.391420 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.391643 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.391702 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:54:16.391916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14332
I20260928 17:54:16.392030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:54:16.392123 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.383821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:35653)
0928 17:54:16.383974 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.384016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.384563 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.384797 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.384801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.385434 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:16.385625 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.385630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.386326 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.386326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.387128 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.387130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.387444 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.387445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.387545 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.388075 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:16.388099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:16.389390 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:16.390706 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.390711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.390713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.390919 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.391106 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.391107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.391107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.391219 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.391424 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:16.391429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:16.391540 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.391619 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.391767 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 17:54:16.392270 392 external_mini_cluster.cc:1428] Setting key 85ef075e534e52939479bdebfd1a7f36
I20260928 17:54:16.392589 14461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:16.392697 14461 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:16.392863 14461 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:16.393254 14136 ts_manager.cc:194] Registered new tserver with Master: b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
I20260928 17:54:16.393620 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35653
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:54:16.416687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:16.501559 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:16.501752 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:16.501775 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:16.502827 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:16.502859 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:16.502883 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:16.504227 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:16.504443 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:16.504688 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.509624 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.509739 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.509874 14466 server_base.cc:1034] running on GCE node
W20260928 17:54:16.509670 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.510155 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.510402 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.511564 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790618056511554 us; error 29 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.514204 14466 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:16.514730 14466 webserver.cc:460] Webserver started at http://127.0.98.3:44741/ using document root <none> and password file <none>
I20260928 17:54:16.514964 14466 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.515043 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.515202 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:16.516431 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "dcc336961489428bb43f8f0c74c19969"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "8279242ea073d446f6a8e652f43719d3"
server_key_iv: "b8a7dedaca816d72cdca9220646a51fe"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.516824 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dcc336961489428bb43f8f0c74c19969"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "8279242ea073d446f6a8e652f43719d3"
server_key_iv: "b8a7dedaca816d72cdca9220646a51fe"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.518134 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:16.518841 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.519026 14466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:16.519119 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "dcc336961489428bb43f8f0c74c19969"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "8279242ea073d446f6a8e652f43719d3"
server_key_iv: "b8a7dedaca816d72cdca9220646a51fe"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.519192 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.529558 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.531765 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.531934 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.532261 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:16.532604 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:16.532662 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.532706 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:16.532732 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.539563 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46139
I20260928 17:54:16.539623 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46139 every 8 connection(s)
I20260928 17:54:16.539997 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:54:16.541399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14466
I20260928 17:54:16.541486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:54:16.541687 392 external_mini_cluster.cc:1428] Setting key a8530e048a59fe6cdc82cc78de1d33f9
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:16.550024 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.540904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50153 (local address 127.0.98.62:45315)
0928 17:54:16.540997 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.541000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.541197 (+ 197us) server_negotiation.cc:407] Connection header received
0928 17:54:16.541812 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.541814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.541827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.541885 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:16.543053 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.543560 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.544666 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.544862 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.544949 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.547132 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:16.547133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:16.547134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:16.547142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:16.548593 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.549034 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.549035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.549036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.549096 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.549360 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.549361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.549361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.549479 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:16.549557 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.549838 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.549919 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 17:54:16.550218 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.540862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:50153)
0928 17:54:16.541067 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.541104 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.541696 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.541967 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.541970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.542733 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:16.542935 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.542940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.543657 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.543658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.544553 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.544556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.544960 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.544962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.545089 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.545663 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:16.545672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:16.547031 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:16.548698 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.548705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.548707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.548942 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.549162 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:16.549163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:16.549164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:16.549280 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:16.549548 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:16.549555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:16.549701 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.549814 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.549976 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":121,"threads_started":1}
I20260928 17:54:16.550760 14595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:16.550868 14595 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:16.551064 14595 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:16.551463 14136 ts_manager.cc:194] Registered new tserver with Master: dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139)
I20260928 17:54:16.551829 14136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50153
I20260928 17:54:16.553314 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:16.561007 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.554682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.0.98.62:45315)
0928 17:54:16.554752 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.554755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.554836 (+ 81us) server_negotiation.cc:407] Connection header received
0928 17:54:16.554876 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.554877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.554885 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.554937 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:16.555483 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.555963 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.556727 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.556934 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.556995 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.558304 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:16.558306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:16.558306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:16.558312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:16.559866 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.560281 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.560282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.560282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.560337 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:16.560530 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:16.560530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:16.560531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:16.560681 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:16.560758 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.560869 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.560936 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:54:16.562966 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:54:16.564767 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:54:16.572038 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.0.1:37060)
0928 17:54:16.567727 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.567775 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.567844 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.568421 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.568422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.568424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:16.568467 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.568468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.569783 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.569784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.571039 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.571042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.571765 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.571766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.571814 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.571825 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.571906 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.571960 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260928 17:54:16.572050 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.1:40817)
0928 17:54:16.568188 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.568193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.568203 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:16.568246 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.568248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.568265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.568337 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:16.568551 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.569664 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.571132 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.571680 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.571771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.571834 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.571927 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.571972 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":99,"threads_started":1}
I20260928 17:54:16.572934 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46235 (local address 127.0.0.1:36458)
0928 17:54:16.567968 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.567977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.568031 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.568280 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.568283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.568286 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:16.568366 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.568368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.570793 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.570794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.571603 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.571605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.572704 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.572705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.572768 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.572772 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.572830 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.572869 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":11,"thread_start_us":102,"threads_started":1}
I20260928 17:54:16.573005 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.98.2:46235)
0928 17:54:16.567924 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.567929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.568060 (+ 131us) server_negotiation.cc:407] Connection header received
0928 17:54:16.568156 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.568159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.568178 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.568273 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:16.568455 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.569659 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.571698 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.572613 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.572708 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.572766 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.572854 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.572895 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260928 17:54:16.573071 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46139 (local address 127.0.0.1:41362)
0928 17:54:16.567860 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.567872 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.567929 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.568747 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.568747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.568749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:16.568797 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.568798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.569624 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.569625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.571041 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.571042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.572808 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.572809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.572877 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.572883 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.572944 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.572986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":43,"thread_start_us":113,"threads_started":1}
I20260928 17:54:16.573279 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.567727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.98.3:46139)
0928 17:54:16.568595 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.568598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.568606 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:54:16.568646 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.568647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.568660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.568747 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:16.568856 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.569522 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.571875 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.572644 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.572712 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.572766 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.573140 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.573194 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":83,"threads_started":1}
I20260928 17:54:16.573220 14261 tablet_service.cc:1467] Processing CreateTablet for tablet 3329a52aaf934a938a8772079202d033 (DEFAULT_TABLE table=test-table [id=9840c16cd4a745479215242e58dc4358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:16.573632 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3329a52aaf934a938a8772079202d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:16.574007 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 3329a52aaf934a938a8772079202d033 (DEFAULT_TABLE table=test-table [id=9840c16cd4a745479215242e58dc4358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:16.574323 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3329a52aaf934a938a8772079202d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:16.576762 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 3329a52aaf934a938a8772079202d033 (DEFAULT_TABLE table=test-table [id=9840c16cd4a745479215242e58dc4358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:16.577078 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3329a52aaf934a938a8772079202d033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:16.577142 14612 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Bootstrap starting.
I20260928 17:54:16.577529 14613 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Bootstrap starting.
I20260928 17:54:16.577934 14612 tablet_bootstrap.cc:654] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:16.578240 14613 tablet_bootstrap.cc:654] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:16.578482 14612 log.cc:824] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:16.578644 14613 log.cc:824] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:16.579283 14612 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: No bootstrap required, opened a new log
I20260928 17:54:16.579341 14612 ts_tablet_manager.cc:1397] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:16.579366 14613 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: No bootstrap required, opened a new log
I20260928 17:54:16.579422 14613 ts_tablet_manager.cc:1397] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:16.580183 14616 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Bootstrap starting.
I20260928 17:54:16.580812 14616 tablet_bootstrap.cc:654] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:16.581308 14616 log.cc:824] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:16.581295 14613 raft_consensus.cc:348] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.581295 14612 raft_consensus.cc:348] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.581476 14613 raft_consensus.cc:374] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:16.581476 14612 raft_consensus.cc:374] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:16.581513 14612 raft_consensus.cc:729] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4369a573873441dca24fb6bbad585ed9, State: Initialized, Role: FOLLOWER
I20260928 17:54:16.581513 14613 raft_consensus.cc:729] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b9bbb2a1ca4254893a25fb1c167371, State: Initialized, Role: FOLLOWER
I20260928 17:54:16.581633 14613 consensus_queue.cc:260] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.581635 14612 consensus_queue.cc:260] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.581908 14612 ts_tablet_manager.cc:1428] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:54:16.581908 14613 ts_tablet_manager.cc:1428] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:54:16.581944 14327 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
I20260928 17:54:16.582031 14461 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
I20260928 17:54:16.582491 14616 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: No bootstrap required, opened a new log
I20260928 17:54:16.582564 14616 ts_tablet_manager.cc:1397] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:16.584060 14616 raft_consensus.cc:348] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.584285 14616 raft_consensus.cc:374] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:16.584333 14616 raft_consensus.cc:729] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc336961489428bb43f8f0c74c19969, State: Initialized, Role: FOLLOWER
I20260928 17:54:16.584441 14616 consensus_queue.cc:260] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.584697 14616 ts_tablet_manager.cc:1428] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:16.584728 14595 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
W20260928 17:54:16.633684 14462 tablet.cc:2369] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:54:16.667265 14618 raft_consensus.cc:482] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:16.667457 14618 raft_consensus.cc:504] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.667891 14618 leader_election.cc:288] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817), dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139)
I20260928 17:54:16.670780 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.668186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53635 (local address 127.0.98.3:46139)
0928 17:54:16.668231 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.668233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.668243 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:16.668286 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.668287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.668298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.668367 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:16.668596 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.669089 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670041 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.670418 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670500 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.670511 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.670670 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.670711 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:54:16.670781 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.668086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46139 (local address 127.0.98.2:53635)
0928 17:54:16.668153 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.668168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.668218 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.668371 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.668371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.668373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:16.668451 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.668453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.669178 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.669179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.669924 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.669926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.670507 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.670593 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.670596 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.670665 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.670709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 17:54:16.671159 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.668255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59789 (local address 127.0.98.1:40817)
0928 17:54:16.668307 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.668310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.668401 (+ 91us) server_negotiation.cc:407] Connection header received
0928 17:54:16.668522 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.668523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.668539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.668625 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:16.668792 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.669291 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670324 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.670735 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.670833 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.671028 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.671070 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:54:16.671151 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3329a52aaf934a938a8772079202d033" candidate_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc336961489428bb43f8f0c74c19969" is_pre_election: true
I20260928 17:54:16.671329 14549 raft_consensus.cc:2454] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b9bbb2a1ca4254893a25fb1c167371 in term 0.
I20260928 17:54:16.671423 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.668114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.98.2:59789)
0928 17:54:16.668312 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.668323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:16.668417 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:16.668631 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:16.668632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:16.668634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:16.668714 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.668715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.669383 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.669384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.670224 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:16.670226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.670845 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.670846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:16.670939 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.670943 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:16.671277 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 17:54:16.671325 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":123,"threads_started":1}
I20260928 17:54:16.671589 14351 leader_election.cc:302] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969; no voters:
I20260928 17:54:16.671746 14618 raft_consensus.cc:2781] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:16.671798 14618 raft_consensus.cc:482] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:16.671821 14618 raft_consensus.cc:3037] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:16.671768 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3329a52aaf934a938a8772079202d033" candidate_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4369a573873441dca24fb6bbad585ed9" is_pre_election: true
I20260928 17:54:16.671947 14281 raft_consensus.cc:2454] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b9bbb2a1ca4254893a25fb1c167371 in term 0.
I20260928 17:54:16.672689 14618 raft_consensus.cc:504] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.672828 14618 leader_election.cc:288] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [CANDIDATE]: Term 1 election: Requested vote from peers 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817), dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139)
I20260928 17:54:16.672983 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3329a52aaf934a938a8772079202d033" candidate_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4369a573873441dca24fb6bbad585ed9"
I20260928 17:54:16.673014 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3329a52aaf934a938a8772079202d033" candidate_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc336961489428bb43f8f0c74c19969"
I20260928 17:54:16.673084 14281 raft_consensus.cc:3037] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:16.673094 14549 raft_consensus.cc:3037] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:16.673804 14549 raft_consensus.cc:2454] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b9bbb2a1ca4254893a25fb1c167371 in term 1.
I20260928 17:54:16.673867 14281 raft_consensus.cc:2454] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b9bbb2a1ca4254893a25fb1c167371 in term 1.
I20260928 17:54:16.673990 14351 leader_election.cc:302] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969; no voters:
I20260928 17:54:16.674077 14618 raft_consensus.cc:2781] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:16.674254 14618 raft_consensus.cc:686] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 LEADER]: Becoming Leader. State: Replica: b3b9bbb2a1ca4254893a25fb1c167371, State: Running, Role: LEADER
I20260928 17:54:16.674386 14618 consensus_queue.cc:237] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:16.675310 14135 catalog_manager.cc:5495] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20260928 17:54:16.692847 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.689993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.98.2:46235)
0928 17:54:16.690061 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:16.690064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:16.690074 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:16.690160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:16.690161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:16.690185 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:16.690271 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:16.690413 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.690915 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.691960 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:16.692335 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:16.692413 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:16.692573 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:16.692676 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:16.692737 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:16.692770 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 17:54:16.692788 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:54:16.694664 14549 raft_consensus.cc:1261] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Refusing update from remote peer b3b9bbb2a1ca4254893a25fb1c167371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:16.694705 14281 raft_consensus.cc:1261] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Refusing update from remote peer b3b9bbb2a1ca4254893a25fb1c167371: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:16.695035 14622 consensus_queue.cc:1035] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:16.695197 14618 consensus_queue.cc:1035] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:16.698387 14627 mvcc.cc:205] Tried to move back new op lower bound from 7334371560218578944 to 7334371560139042816. Current Snapshot: MvccSnapshot[applied={T|T < 7334371560218578944}]
I20260928 17:54:16.699899 14626 mvcc.cc:205] Tried to move back new op lower bound from 7334371560218578944 to 7334371560139042816. Current Snapshot: MvccSnapshot[applied={T|T < 7334371560218578944 or (T in {7334371560218578944})}]
I20260928 17:54:16.700067 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:16.700227 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:16.701009 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Sending DeleteTablet for 3 replicas of tablet 3329a52aaf934a938a8772079202d033
I20260928 17:54:16.701349 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 3329a52aaf934a938a8772079202d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36458
I20260928 17:54:16.701370 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 3329a52aaf934a938a8772079202d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41362
I20260928 17:54:16.701462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14198
I20260928 17:54:16.701572 14261 tablet_service.cc:1514] Processing DeleteTablet for tablet 3329a52aaf934a938a8772079202d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37060
I20260928 17:54:16.702075 14635 tablet_replica.cc:330] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: stopping tablet replica
I20260928 17:54:16.702164 14635 raft_consensus.cc:2229] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:16.702260 14635 raft_consensus.cc:2258] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:16.702402 14634 tablet_replica.cc:330] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: stopping tablet replica
I20260928 17:54:16.702481 14634 raft_consensus.cc:2229] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:54:16.702564 14634 raft_consensus.cc:2258] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:16.702971 14634 ts_tablet_manager.cc:1905] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:16.704986 14634 ts_tablet_manager.cc:1918] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:16.705075 14634 log.cc:1196] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/3329a52aaf934a938a8772079202d033
I20260928 17:54:16.705402 14634 ts_tablet_manager.cc:1939] T 3329a52aaf934a938a8772079202d033 P b3b9bbb2a1ca4254893a25fb1c167371: Deleting consensus metadata
I20260928 17:54:16.706110 14122 catalog_manager.cc:4841] TS b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): tablet 3329a52aaf934a938a8772079202d033 (table test-table [id=9840c16cd4a745479215242e58dc4358]) successfully deleted
W20260928 17:54:16.707791 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40817 (error 108)
I20260928 17:54:16.708196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14332
I20260928 17:54:16.708658 14635 ts_tablet_manager.cc:1905] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:16.708964 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.708245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.0.1:37066)
0928 17:54:16.708880 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.708935 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611}
W20260928 17:54:16.709132 14120 catalog_manager.cc:4584] TS 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3329a52aaf934a938a8772079202d033: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40817: connect: Connection refused (error 111)
I20260928 17:54:16.710175 14635 ts_tablet_manager.cc:1918] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:16.710237 14635 log.cc:1196] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/3329a52aaf934a938a8772079202d033
I20260928 17:54:16.710496 14635 ts_tablet_manager.cc:1939] T 3329a52aaf934a938a8772079202d033 P dcc336961489428bb43f8f0c74c19969: Deleting consensus metadata
I20260928 17:54:16.710904 14123 catalog_manager.cc:4841] TS dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): tablet 3329a52aaf934a938a8772079202d033 (table test-table [id=9840c16cd4a745479215242e58dc4358]) successfully deleted
I20260928 17:54:16.715341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14466
I20260928 17:54:16.720580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14104
I20260928 17:54:16.725829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--webserver_interface=127.0.98.62
--webserver_port=35549
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:16.807614 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:16.807807 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:16.807832 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:16.808923 14637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:16.808959 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:16.808972 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:16.808984 14637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:16.808996 14637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:16.810362 14637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:16.810551 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:16.810712 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.819144 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.819169 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.819381 14637 server_base.cc:1034] running on GCE node
W20260928 17:54:16.819154 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.819742 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.820114 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.821298 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790618056821279 us; error 42 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.835526 14637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:16.836078 14637 webserver.cc:460] Webserver started at http://127.0.98.62:35549/ using document root <none> and password file <none>
I20260928 17:54:16.836410 14637 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.836511 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.837988 14637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:54:16.838673 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.838855 14637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:54:16.838961 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "fca03e5923ae4b3a843dc9c6fcd226aa"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "dca8b79b3b2c6d4fe698622cfd067383"
server_key_iv: "8430a7623d55f026856e6a657b5476e9"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.839377 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.847322 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.849400 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.849622 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.853834 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45315
I20260928 17:54:16.853873 14705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45315 every 8 connection(s)
I20260928 17:54:16.854336 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:16.856007 14706 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:54:16.856763 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap starting.
I20260928 17:54:16.860926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14637
I20260928 17:54:16.861315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=36795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:16.866617 14706 log.cc:824] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:16.868984 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:16.869344 14706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap complete.
I20260928 17:54:16.871831 14706 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.872152 14706 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Initialized, Role: FOLLOWER
I20260928 17:54:16.872376 14706 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.872473 14706 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:16.872539 14706 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:16.872593 14706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:54:16.873737 14706 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.873871 14706 leader_election.cc:302] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca03e5923ae4b3a843dc9c6fcd226aa; no voters:
I20260928 17:54:16.874112 14706 leader_election.cc:288] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:54:16.874155 14710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:54:16.874290 14710 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 LEADER]: Becoming Leader. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Running, Role: LEADER
I20260928 17:54:16.874382 14710 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:16.874413 14706 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:16.874702 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca03e5923ae4b3a843dc9c6fcd226aa. Latest consensus state: current_term: 2 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:16.874778 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:16.875036 14714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:16.875962 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:16.876137 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:16.876555 14714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:16.876678 14714 catalog_manager.cc:1259] Loaded cluster ID: 6b43db108f394267a73cee536ff88b6d
I20260928 17:54:16.876943 14714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:16.877223 14714 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 17:54:16.881618 14714 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 17:54:16.884234 14714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:16.884457 14714 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Loaded TSK: 0
I20260928 17:54:16.884694 14714 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:54:16.945688 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:16.945935 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:16.945976 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:16.947170 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:16.947245 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:16.947291 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:16.948783 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36795
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:16.949067 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:16.949309 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:16.954401 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.954417 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:16.954453 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:16.954857 14708 server_base.cc:1034] running on GCE node
I20260928 17:54:16.955106 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:16.955384 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:16.956547 14708 hybrid_clock.cc:648] HybridClock initialized: now 1790618056956531 us; error 28 us; skew 500 ppm
Sep 28 17:54:16 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:16.959394 14708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:16.960000 14708 webserver.cc:460] Webserver started at http://127.0.98.1:36795/ using document root <none> and password file <none>
I20260928 17:54:16.960295 14708 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:16.960381 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:16.962275 14708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:16.963121 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:16.963351 14708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:16.963464 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "4369a573873441dca24fb6bbad585ed9"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "9b5b80ae5d787b56f55b15d8483789f4"
server_key_iv: "c5d0ddf4b248cccc6ce3ea6114b26595"
server_key_version: "encryptionkey@0"
I20260928 17:54:16.963929 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:16.983189 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:16.985800 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:16.985965 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:16.986245 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:16.986759 14750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:54:16.988289 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:54:16.988363 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:54:16.988404 14708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:54:16.989276 14708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:54:16.989328 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:16.989389 14750 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Bootstrap starting.
I20260928 17:54:16.997313 14708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40817
I20260928 17:54:16.997790 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:16.999723 14750 log.cc:824] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:17.000013 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40817 every 8 connection(s)
I20260928 17:54:17.002362 14750 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:17.002653 14750 tablet_bootstrap.cc:492] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Bootstrap complete.
I20260928 17:54:17.003623 14750 ts_tablet_manager.cc:1397] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260928 17:54:17.005023 14750 raft_consensus.cc:348] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:17.005275 14750 raft_consensus.cc:729] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4369a573873441dca24fb6bbad585ed9, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.005455 14750 consensus_queue.cc:260] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } }
I20260928 17:54:17.005771 14750 ts_tablet_manager.cc:1428] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260928 17:54:17.007272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14708
I20260928 17:54:17.007570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46235
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=36129
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:17.012413 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.998535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:59077)
0928 17:54:16.999919 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:16.999966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.000702 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.001245 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.001250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.001606 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:17.001873 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.001881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.003579 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.003580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.004854 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.004856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.005826 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.005827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.005968 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.006563 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:17.006571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:17.008332 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:17.010808 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.010815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.010817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.011044 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.011356 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.011357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.011357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.011452 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.011883 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:17.011887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:17.011999 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.012074 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.012265 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":219,"threads_started":1}
I20260928 17:54:17.012579 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:16.998627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59077 (local address 127.0.98.62:45315)
0928 17:54:17.000306 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.000315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.000353 (+ 38us) server_negotiation.cc:407] Connection header received
0928 17:54:17.000961 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.000969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.001030 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.001190 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:17.002023 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.003432 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.004964 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.005729 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.005910 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.008466 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:17.008472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:17.008472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:17.008486 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:17.010710 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.011145 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.011149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.011149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.011289 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.011525 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.011526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.011527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.011815 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:17.011902 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.012232 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.012364 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":92,"threads_started":1}
I20260928 17:54:17.013168 14858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:17.013263 14858 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:17.013448 14858 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:17.014258 14670 ts_manager.cc:194] Registered new tserver with Master: 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817)
I20260928 17:54:17.015602 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59077
I20260928 17:54:17.016624 14858 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
I20260928 17:54:17.027354 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.015229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.98.1:40817)
0928 17:54:17.022773 (+ 7544us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.022779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.022792 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:54:17.022856 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.022859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.022878 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.022999 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:17.023418 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.024655 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.026255 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.026896 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.026972 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.027030 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.027232 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.027275 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7478,"thread_start_us":70,"threads_started":1}
I20260928 17:54:17.027369 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.014977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.0.1:37074)
0928 17:54:17.015956 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.015997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.016075 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.023188 (+ 7113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.023193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.023197 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:17.023321 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.023324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.024743 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.024744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.026142 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.026147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.027003 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.027004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.027100 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.027116 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.027210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.027269 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":95,"threads_started":1}
I20260928 17:54:17.028147 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 3329a52aaf934a938a8772079202d033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37074
I20260928 17:54:17.028785 14867 tablet_replica.cc:330] stopping tablet replica
I20260928 17:54:17.028911 14867 raft_consensus.cc:2229] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:17.028995 14867 raft_consensus.cc:2258] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:17.029409 14867 ts_tablet_manager.cc:1905] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:17.031776 14867 ts_tablet_manager.cc:1918] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:17.031869 14867 log.cc:1196] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/3329a52aaf934a938a8772079202d033
I20260928 17:54:17.032169 14867 ts_tablet_manager.cc:1939] T 3329a52aaf934a938a8772079202d033 P 4369a573873441dca24fb6bbad585ed9: Deleting consensus metadata
I20260928 17:54:17.032793 14654 catalog_manager.cc:4841] TS 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817): tablet 3329a52aaf934a938a8772079202d033 (table test-table [id=9840c16cd4a745479215242e58dc4358]) successfully deleted
W20260928 17:54:17.097826 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.098055 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.098080 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.099179 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.099222 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.099251 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:17.100827 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36129
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.101052 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.101243 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:54:17.106494 14864 server_base.cc:1034] running on GCE node
W20260928 17:54:17.106603 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.106523 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.106523 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.106976 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.107225 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.108405 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790618057108391 us; error 31 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.110997 14864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:17.111423 14864 webserver.cc:460] Webserver started at http://127.0.98.2:36129/ using document root <none> and password file <none>
I20260928 17:54:17.111665 14864 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.111745 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.113108 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.113713 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.113960 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.114074 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "afc52d74796478b9be5397c1d730551c"
server_key_iv: "8a3fe4f4647c67d9587711705b7cde91"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.114425 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.147007 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.149187 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.149356 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.149607 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:17.149909 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:17.149966 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.150007 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:17.150036 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.156308 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46235
I20260928 17:54:17.156349 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46235 every 8 connection(s)
I20260928 17:54:17.156689 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:17.165963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14864
I20260928 17:54:17.166294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46139
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=44741
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:17.166342 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.157394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48191 (local address 127.0.98.62:45315)
0928 17:54:17.157459 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.157462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.157677 (+ 215us) server_negotiation.cc:407] Connection header received
0928 17:54:17.158168 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.158169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.158178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.158230 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:17.159189 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.159681 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.160795 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.160998 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.161070 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.163400 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:17.163401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:17.163401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:17.163408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:17.164811 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.165244 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.165245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.165246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.165300 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.165558 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.165560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.165560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.165723 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:17.165791 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.166194 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.166273 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:54:17.166618 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.157397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:48191)
0928 17:54:17.157548 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.157587 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.158081 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.158286 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.158289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.158889 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:17.159096 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.159101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.159769 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.159770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.160718 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.160720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.161082 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.161083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.161208 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.161820 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:17.161829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:17.163307 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:17.164899 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.164905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.164907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.165161 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.165361 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.165361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.165362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.165465 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.165816 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:17.165824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:17.166030 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.166176 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.166364 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260928 17:54:17.167250 14996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:17.167369 14996 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:17.167567 14996 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:17.168027 14670 ts_manager.cc:194] Registered new tserver with Master: b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
I20260928 17:54:17.168603 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48191
W20260928 17:54:17.247764 14999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.247968 14999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.247990 14999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.249154 14999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.249197 14999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.249222 14999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:17.250725 14999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44741
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.250949 14999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.251168 14999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:17.256542 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.256528 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.256528 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.257026 14999 server_base.cc:1034] running on GCE node
I20260928 17:54:17.257229 14999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.257444 14999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.258632 14999 hybrid_clock.cc:648] HybridClock initialized: now 1790618057258618 us; error 27 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.261183 14999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:17.261696 14999 webserver.cc:460] Webserver started at http://127.0.98.3:44741/ using document root <none> and password file <none>
I20260928 17:54:17.261917 14999 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.261996 14999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.263373 14999 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.264002 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.264191 14999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:17.264292 14999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "dcc336961489428bb43f8f0c74c19969"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "8279242ea073d446f6a8e652f43719d3"
server_key_iv: "b8a7dedaca816d72cdca9220646a51fe"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.264695 14999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.283670 14999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.286119 14999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.286291 14999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.286540 14999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:17.286861 14999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:17.286916 14999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.286960 14999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:17.286989 14999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.292896 14999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46139
I20260928 17:54:17.292985 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46139 every 8 connection(s)
I20260928 17:54:17.293346 14999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:17.302322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 14999
I20260928 17:54:17.303690 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.294224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38621 (local address 127.0.98.62:45315)
0928 17:54:17.294300 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.294302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.294527 (+ 225us) server_negotiation.cc:407] Connection header received
0928 17:54:17.295107 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.295108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.295119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.295179 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:17.296222 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.296743 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.297797 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.298017 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.298118 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.300460 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:17.300461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:17.300461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:17.300470 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:17.301771 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.302471 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.302472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.302472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.302586 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.302955 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.302956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.302956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.303115 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:17.303180 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.303557 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.303627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:54:17.304035 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.294212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:38621)
0928 17:54:17.294398 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.294435 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.294986 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.295259 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.295263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.295871 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:17.296109 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.296116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.296857 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.296858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.297697 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.297699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.298128 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.298130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.298262 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.298857 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:17.298866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:17.300350 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:17.301859 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.301866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.301869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.302107 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.302727 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:17.302727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:17.302728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:17.302869 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:17.303185 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:17.303192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:17.303378 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.303510 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.303703 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260928 17:54:17.304540 15128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45315
I20260928 17:54:17.304665 15128 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:17.304853 15128 heartbeater.cc:511] Master 127.0.98.62:45315 requested a full tablet report, sending...
I20260928 17:54:17.305269 14670 ts_manager.cc:194] Registered new tserver with Master: dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139)
I20260928 17:54:17.305724 14670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38621
I20260928 17:54:17.309388 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.302748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.62:45315)
0928 17:54:17.302881 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.302882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.302941 (+ 59us) server_negotiation.cc:407] Connection header received
0928 17:54:17.302999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.303000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.303011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.303095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:17.303851 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.304549 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.305561 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.305857 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.305922 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.307149 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:17.307151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:17.307151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:17.307157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:17.308447 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.308776 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.308777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.308777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.308826 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:17.308989 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:17.308989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:17.308989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:17.309087 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:17.309208 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.309240 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.309298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 17:54:17.310122 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:17.312729 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.310488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.0.98.62:45315)
0928 17:54:17.310523 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.310525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.310660 (+ 135us) server_negotiation.cc:407] Connection header received
0928 17:54:17.310695 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.310695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.310726 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.310786 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:17.310861 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.311308 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.312023 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.312272 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.312326 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.312452 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:17.312548 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:17.312608 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.312620 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.312637 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 17:54:17.313023 14670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:54:17.314678 14670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:54:17.317030 14792 tablet_service.cc:1467] Processing CreateTablet for tablet cd777fbd075d479185233d45ebc2b75b (DEFAULT_TABLE table=test-table [id=d04de6e960db4676987fdad58361c5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:17.317293 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd777fbd075d479185233d45ebc2b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:17.319029 14750 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Bootstrap starting.
I20260928 17:54:17.319299 14750 tablet_bootstrap.cc:654] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:17.319958 14750 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: No bootstrap required, opened a new log
I20260928 17:54:17.319999 14750 ts_tablet_manager.cc:1397] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:17.320150 14750 raft_consensus.cc:348] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.320230 14750 raft_consensus.cc:374] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:17.320281 14750 raft_consensus.cc:729] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4369a573873441dca24fb6bbad585ed9, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.320333 14750 consensus_queue.cc:260] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.320428 14750 ts_tablet_manager.cc:1428] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:17.320986 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.316758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46235 (local address 127.0.0.1:36476)
0928 17:54:17.316912 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.316923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.316972 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.317486 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.317487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.317490 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:17.317546 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.317547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.318586 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.318587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.319598 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.319600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.320741 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.320742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.320807 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.320811 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.320868 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.320913 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260928 17:54:17.321218 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.317104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.0.98.2:46235)
0928 17:54:17.317306 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.317311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.317328 (+ 17us) server_negotiation.cc:407] Connection header received
0928 17:54:17.317380 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.317382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.317400 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.317478 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:17.317624 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.318485 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.319695 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.320643 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.320731 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.320798 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.320889 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.320938 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260928 17:54:17.321321 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.316701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46139 (local address 127.0.0.1:41374)
0928 17:54:17.316748 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.316758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.316794 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.317638 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.317639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.317641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:17.317680 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.317681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.318985 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.318986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.319993 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.319995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.321096 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.321097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.321160 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.321164 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.321211 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.321262 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:54:17.321594 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.316884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.0.98.3:46139)
0928 17:54:17.317321 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.317326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.317337 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:54:17.317379 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.317381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.317399 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.317756 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:17.317831 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.318874 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.320117 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.320938 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.321002 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.321059 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.321453 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.321507 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":89,"threads_started":1}
I20260928 17:54:17.322083 14930 tablet_service.cc:1467] Processing CreateTablet for tablet cd777fbd075d479185233d45ebc2b75b (DEFAULT_TABLE table=test-table [id=d04de6e960db4676987fdad58361c5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:17.322146 15062 tablet_service.cc:1467] Processing CreateTablet for tablet cd777fbd075d479185233d45ebc2b75b (DEFAULT_TABLE table=test-table [id=d04de6e960db4676987fdad58361c5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:17.322451 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd777fbd075d479185233d45ebc2b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:17.322451 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd777fbd075d479185233d45ebc2b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:17.326231 15139 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Bootstrap starting.
I20260928 17:54:17.326231 15138 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Bootstrap starting.
I20260928 17:54:17.326968 15138 tablet_bootstrap.cc:654] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:17.326968 15139 tablet_bootstrap.cc:654] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:17.327595 15138 log.cc:824] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:17.327620 15139 log.cc:824] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:17.328450 15139 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: No bootstrap required, opened a new log
I20260928 17:54:17.328553 15139 ts_tablet_manager.cc:1397] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:17.328634 15138 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: No bootstrap required, opened a new log
I20260928 17:54:17.328708 15138 ts_tablet_manager.cc:1397] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:54:17.329931 15139 raft_consensus.cc:348] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.330106 15139 raft_consensus.cc:374] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:17.330150 15139 raft_consensus.cc:729] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc336961489428bb43f8f0c74c19969, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.330037 15138 raft_consensus.cc:348] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.330202 15138 raft_consensus.cc:374] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:17.330248 15138 raft_consensus.cc:729] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b9bbb2a1ca4254893a25fb1c167371, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.330282 15139 consensus_queue.cc:260] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.330366 15138 consensus_queue.cc:260] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.330598 15128 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
I20260928 17:54:17.330664 15138 ts_tablet_manager.cc:1428] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:17.330718 14996 heartbeater.cc:503] Master 127.0.98.62:45315 was elected leader, sending a full tablet report...
I20260928 17:54:17.330600 15139 ts_tablet_manager.cc:1428] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:54:17.345669 15142 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:17.345803 15142 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.346170 15142 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235), 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817)
I20260928 17:54:17.350423 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.346476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49119 (local address 127.0.98.2:46235)
0928 17:54:17.346531 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.346533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.346541 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:54:17.346611 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.346612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.346624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.346723 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:17.346916 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.347754 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.349380 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.350037 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.350114 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.350129 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.350295 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.350344 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:54:17.350423 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.346472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52051 (local address 127.0.98.1:40817)
0928 17:54:17.346522 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.346525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.346669 (+ 144us) server_negotiation.cc:407] Connection header received
0928 17:54:17.346701 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.346702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.346715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.346776 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:17.346907 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.347734 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.349379 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.349993 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.350132 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.350144 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.350295 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.350342 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:54:17.350443 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.346381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46235 (local address 127.0.98.3:49119)
0928 17:54:17.346439 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.346452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.346494 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.346741 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.346742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.346743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:17.346838 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.346840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.347847 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.347848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.349266 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.349271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.350119 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.350120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.350200 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.350206 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.350268 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.350324 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":88}
I20260928 17:54:17.350450 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.346398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.98.3:52051)
0928 17:54:17.346600 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.346607 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.346651 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.346784 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:17.346785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:17.346786 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:17.346838 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.346840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.347846 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.347848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.349266 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:17.349271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.350141 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.350142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:17.350212 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.350216 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:17.350268 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 17:54:17.350323 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":65,"thread_start_us":99,"threads_started":1}
I20260928 17:54:17.350936 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd777fbd075d479185233d45ebc2b75b" candidate_uuid: "dcc336961489428bb43f8f0c74c19969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" is_pre_election: true
I20260928 17:54:17.351055 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd777fbd075d479185233d45ebc2b75b" candidate_uuid: "dcc336961489428bb43f8f0c74c19969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4369a573873441dca24fb6bbad585ed9" is_pre_election: true
I20260928 17:54:17.351146 14950 raft_consensus.cc:2454] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc336961489428bb43f8f0c74c19969 in term 0.
I20260928 17:54:17.351195 14812 raft_consensus.cc:2454] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcc336961489428bb43f8f0c74c19969 in term 0.
I20260928 17:54:17.351397 15017 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969; no voters:
I20260928 17:54:17.351557 15142 raft_consensus.cc:2781] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:17.351624 15142 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:17.351642 15142 raft_consensus.cc:3037] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:17.352459 15142 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.352622 15142 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [CANDIDATE]: Term 1 election: Requested vote from peers b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235), 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817)
I20260928 17:54:17.352753 14812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd777fbd075d479185233d45ebc2b75b" candidate_uuid: "dcc336961489428bb43f8f0c74c19969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4369a573873441dca24fb6bbad585ed9"
I20260928 17:54:17.352780 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd777fbd075d479185233d45ebc2b75b" candidate_uuid: "dcc336961489428bb43f8f0c74c19969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
I20260928 17:54:17.352835 14812 raft_consensus.cc:3037] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:17.352885 14950 raft_consensus.cc:3037] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:17.353619 14812 raft_consensus.cc:2454] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc336961489428bb43f8f0c74c19969 in term 1.
I20260928 17:54:17.353790 15015 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9, dcc336961489428bb43f8f0c74c19969; no voters:
I20260928 17:54:17.353797 14950 raft_consensus.cc:2454] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcc336961489428bb43f8f0c74c19969 in term 1.
I20260928 17:54:17.353917 15142 raft_consensus.cc:2781] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:17.354108 15142 raft_consensus.cc:686] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 1 LEADER]: Becoming Leader. State: Replica: dcc336961489428bb43f8f0c74c19969, State: Running, Role: LEADER
I20260928 17:54:17.354226 15142 consensus_queue.cc:237] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.355098 14670 catalog_manager.cc:5495] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcc336961489428bb43f8f0c74c19969 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dcc336961489428bb43f8f0c74c19969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } health_report { overall_health: UNKNOWN } } }
I20260928 17:54:17.366772 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.364037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.3:46139)
0928 17:54:17.364111 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:17.364114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:17.364143 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:54:17.364241 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:17.364242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:17.364265 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:17.364360 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:17.364485 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.365030 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.365865 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:17.366097 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:17.366183 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:17.366373 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:17.366495 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:17.366604 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:17.366633 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 17:54:17.366654 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:54:17.368804 14950 raft_consensus.cc:1261] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Refusing update from remote peer dcc336961489428bb43f8f0c74c19969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:17.368794 14812 raft_consensus.cc:1261] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Refusing update from remote peer dcc336961489428bb43f8f0c74c19969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:17.369176 15142 consensus_queue.cc:1035] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:17.369328 15145 consensus_queue.cc:1035] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:17.372408 15150 mvcc.cc:205] Tried to move back new op lower bound from 7334371562979172352 to 7334371562923728896. Current Snapshot: MvccSnapshot[applied={T|T < 7334371562979172352}]
I20260928 17:54:17.372534 15149 mvcc.cc:205] Tried to move back new op lower bound from 7334371562979172352 to 7334371562923728896. Current Snapshot: MvccSnapshot[applied={T|T < 7334371562979172352 or (T in {7334371562979172352})}]
I20260928 17:54:17.375099 14669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:17.375236 14669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:17.376046 14669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Sending DeleteTablet for 3 replicas of tablet cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:17.376426 15062 tablet_service.cc:1514] Processing DeleteTablet for tablet cd777fbd075d479185233d45ebc2b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41374
I20260928 17:54:17.376535 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet cd777fbd075d479185233d45ebc2b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36476
I20260928 17:54:17.376574 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet cd777fbd075d479185233d45ebc2b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37074
I20260928 17:54:17.376662 14867 tablet_replica.cc:330] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: stopping tablet replica
I20260928 17:54:17.376726 14867 raft_consensus.cc:2229] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:17.376792 14867 raft_consensus.cc:2258] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:17.376881 15157 tablet_replica.cc:330] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: stopping tablet replica
I20260928 17:54:17.377038 15157 raft_consensus.cc:2229] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:54:17.377043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14708
I20260928 17:54:17.377223 15157 raft_consensus.cc:2258] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:17.378168 15157 ts_tablet_manager.cc:1905] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:17.380697 15158 tablet_replica.cc:330] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: stopping tablet replica
I20260928 17:54:17.380812 15158 raft_consensus.cc:2229] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:17.380908 15158 raft_consensus.cc:2258] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:17.381402 15158 ts_tablet_manager.cc:1905] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:17.383239 15158 ts_tablet_manager.cc:1918] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:17.383373 15158 log.cc:1196] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:17.383781 15158 ts_tablet_manager.cc:1939] T cd777fbd075d479185233d45ebc2b75b P b3b9bbb2a1ca4254893a25fb1c167371: Deleting consensus metadata
I20260928 17:54:17.384255 14656 catalog_manager.cc:4841] TS b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): tablet cd777fbd075d479185233d45ebc2b75b (table test-table [id=d04de6e960db4676987fdad58361c5eb]) successfully deleted
I20260928 17:54:17.384490 15157 ts_tablet_manager.cc:1918] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:17.384554 15157 log.cc:1196] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:17.384773 15157 ts_tablet_manager.cc:1939] T cd777fbd075d479185233d45ebc2b75b P dcc336961489428bb43f8f0c74c19969: Deleting consensus metadata
I20260928 17:54:17.385217 14657 catalog_manager.cc:4841] TS dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): tablet cd777fbd075d479185233d45ebc2b75b (table test-table [id=d04de6e960db4676987fdad58361c5eb]) successfully deleted
W20260928 17:54:17.388135 14654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40817 (error 108)
I20260928 17:54:17.388621 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.388495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40817 (local address 127.0.0.1:37084)
0928 17:54:17.388548 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.388583 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:54:17.388712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14864
W20260928 17:54:17.390879 14654 catalog_manager.cc:4584] TS 4369a573873441dca24fb6bbad585ed9 (127.0.98.1:40817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd777fbd075d479185233d45ebc2b75b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40817: connect: Connection refused (error 111)
I20260928 17:54:17.394433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14999
I20260928 17:54:17.399626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 14637
I20260928 17:54:17.405038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--webserver_interface=127.0.98.62
--webserver_port=35549
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:17.487105 15160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.487349 15160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.487386 15160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.488591 15160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:17.488656 15160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.488680 15160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.488701 15160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:17.488731 15160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:17.490221 15160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.490491 15160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.490711 15160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:17.495599 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.495599 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.495865 15160 server_base.cc:1034] running on GCE node
W20260928 17:54:17.495639 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.496217 15160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.496552 15160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.497713 15160 hybrid_clock.cc:648] HybridClock initialized: now 1790618057497694 us; error 31 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.500429 15160 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:17.500862 15160 webserver.cc:460] Webserver started at http://127.0.98.62:35549/ using document root <none> and password file <none>
I20260928 17:54:17.501085 15160 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.501170 15160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.502472 15160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.503100 15175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.503269 15160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:17.503365 15160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "fca03e5923ae4b3a843dc9c6fcd226aa"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "dca8b79b3b2c6d4fe698622cfd067383"
server_key_iv: "8430a7623d55f026856e6a657b5476e9"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.503712 15160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.520895 15160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.522895 15160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.523056 15160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.527256 15160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45315
I20260928 17:54:17.527284 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45315 every 8 connection(s)
I20260928 17:54:17.527679 15160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:17.529477 15228 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:54:17.529546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15160
I20260928 17:54:17.529850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40817
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=36795
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:17.530294 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap starting.
I20260928 17:54:17.539546 15228 log.cc:824] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:17.542071 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:17.542415 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa: Bootstrap complete.
I20260928 17:54:17.544834 15228 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:17.545092 15228 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.545327 15228 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:17.545441 15228 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:17.545496 15228 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:17.545563 15228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:54:17.546572 15228 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:17.546701 15228 leader_election.cc:302] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca03e5923ae4b3a843dc9c6fcd226aa; no voters:
I20260928 17:54:17.546921 15228 leader_election.cc:288] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 17:54:17.546967 15232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 3 FOLLOWER]: Leader election won for term 3
I20260928 17:54:17.547101 15232 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [term 3 LEADER]: Becoming Leader. State: Replica: fca03e5923ae4b3a843dc9c6fcd226aa, State: Running, Role: LEADER
I20260928 17:54:17.547192 15232 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } }
I20260928 17:54:17.547232 15228 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:17.547461 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca03e5923ae4b3a843dc9c6fcd226aa. Latest consensus state: current_term: 3 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:17.547540 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:17.547730 15236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:17.548681 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca03e5923ae4b3a843dc9c6fcd226aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45315 } } }
I20260928 17:54:17.548696 15236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:17.549217 15236 catalog_manager.cc:1259] Loaded cluster ID: 6b43db108f394267a73cee536ff88b6d
I20260928 17:54:17.549291 15236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:17.548867 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca03e5923ae4b3a843dc9c6fcd226aa [sys.catalog]: This master's current role is: LEADER
F20260928 17:54:17.549548 15236 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fba08cd89ad google::LogMessage::Fail() at ??:0
@ 0x7fba08cdb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fba08cd84de google::LogMessage::Flush() at ??:0
@ 0x7fba08cd9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fba10bfd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fba10c068b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fba0aba60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fba0ab9b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fba0b9ff6db start_thread at ??:0
@ 0x7fba077d661f clone at ??:0
W20260928 17:54:17.630482 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.630700 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.630724 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.631908 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.631953 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.631979 15230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:17.633492 15230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36795
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.633751 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.634009 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:17.639096 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.639240 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.639293 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.639706 15230 server_base.cc:1034] running on GCE node
I20260928 17:54:17.639933 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.640257 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.641413 15230 hybrid_clock.cc:648] HybridClock initialized: now 1790618057641393 us; error 38 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.643925 15230 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:17.644428 15230 webserver.cc:460] Webserver started at http://127.0.98.1:36795/ using document root <none> and password file <none>
I20260928 17:54:17.644683 15230 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.644755 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.646174 15230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:54:17.646927 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.647133 15230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:54:17.647236 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "4369a573873441dca24fb6bbad585ed9"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "9b5b80ae5d787b56f55b15d8483789f4"
server_key_iv: "c5d0ddf4b248cccc6ce3ea6114b26595"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.647627 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.666153 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.668411 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.668587 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.668828 15230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:17.669245 15281 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:54:17.671257 15230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:54:17.671347 15230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:54:17.671412 15230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:54:17.672138 15230 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:54:17.672202 15230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.672233 15281 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Bootstrap starting.
I20260928 17:54:17.679361 15230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40817
I20260928 17:54:17.679540 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40817 every 8 connection(s)
I20260928 17:54:17.679806 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:17.683441 15281 log.cc:824] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:17.685496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15230
I20260928 17:54:17.685791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46235
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=36129
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:17.686256 15281 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:17.686558 15281 tablet_bootstrap.cc:492] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Bootstrap complete.
I20260928 17:54:17.687800 15281 ts_tablet_manager.cc:1397] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.011s
I20260928 17:54:17.689637 15281 raft_consensus.cc:348] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.690061 15281 raft_consensus.cc:729] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4369a573873441dca24fb6bbad585ed9, State: Initialized, Role: FOLLOWER
I20260928 17:54:17.690222 15281 consensus_queue.cc:260] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:17.690527 15281 ts_tablet_manager.cc:1428] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260928 17:54:17.758751 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:54:17.680561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:53211)
0928 17:54:17.680775 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.680814 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:17.681262 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:17.758518 (+ 77256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
W20260928 17:54:17.759049 15389 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45315 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45315: Client connection negotiation failed: client connection to 127.0.98.62:45315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:54:17.759388 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.759267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:38577)
0928 17:54:17.759324 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.759369 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:54:17.759761 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.759677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:50691)
0928 17:54:17.759725 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.759747 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 17:54:17.759892 15389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:54:17.771441 15393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.771703 15393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.771745 15393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.773072 15393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.773144 15393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.773196 15393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:17.774749 15393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36129
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.775034 15393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.775286 15393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:17.780488 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.780455 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.780512 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.780845 15393 server_base.cc:1034] running on GCE node
I20260928 17:54:17.781018 15393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.781227 15393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.782372 15393 hybrid_clock.cc:648] HybridClock initialized: now 1790618057782366 us; error 35 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.785104 15393 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:17.785625 15393 webserver.cc:460] Webserver started at http://127.0.98.2:36129/ using document root <none> and password file <none>
I20260928 17:54:17.785853 15393 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.785921 15393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.787359 15393 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.788066 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.788436 15393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:17.788564 15393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "b3b9bbb2a1ca4254893a25fb1c167371"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "afc52d74796478b9be5397c1d730551c"
server_key_iv: "8a3fe4f4647c67d9587711705b7cde91"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.788930 15393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.798647 15393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.801005 15393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.801165 15393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.801388 15393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:17.801682 15393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:17.801736 15393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.801776 15393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:17.801805 15393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.807766 15393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46235
I20260928 17:54:17.807821 15522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46235 every 8 connection(s)
I20260928 17:54:17.808204 15393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:54:17.809244 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.808868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:33647)
0928 17:54:17.809017 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.809102 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260928 17:54:17.809684 15523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45315 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45315: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
I20260928 17:54:17.810040 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.809909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:56403)
0928 17:54:17.809985 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.810017 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 17:54:17.810349 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.810280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:52873)
0928 17:54:17.810311 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.810333 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 17:54:17.810465 15523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:54:17.811714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15393
I20260928 17:54:17.811996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46139
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45315
--webserver_port=44741
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:17.894001 15526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:17.894193 15526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:17.894215 15526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:17.895254 15526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:17.895287 15526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:17.895313 15526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:17.896663 15526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44741
--tserver_master_addrs=127.0.98.62:45315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:17.896867 15526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:17.897032 15526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:17.902057 15532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.902251 15526 server_base.cc:1034] running on GCE node
W20260928 17:54:17.902067 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:17.902067 15531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:17.902630 15526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:17.902915 15526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:17.904129 15526 hybrid_clock.cc:648] HybridClock initialized: now 1790618057904085 us; error 39 us; skew 500 ppm
Sep 28 17:54:17 dist-test-slave-zqlt krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:17.906775 15526 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:17.907187 15526 webserver.cc:460] Webserver started at http://127.0.98.3:44741/ using document root <none> and password file <none>
I20260928 17:54:17.907415 15526 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:17.907459 15526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:17.908855 15526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:17.909550 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.909783 15526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:54:17.909893 15526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "dcc336961489428bb43f8f0c74c19969"
format_stamp: "Formatted at 2026-09-28 17:54:16 on dist-test-slave-zqlt"
server_key: "8279242ea073d446f6a8e652f43719d3"
server_key_iv: "b8a7dedaca816d72cdca9220646a51fe"
server_key_version: "encryptionkey@0"
I20260928 17:54:17.910245 15526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:17.917595 15526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:17.919657 15526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:17.919823 15526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:17.920071 15526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:17.920431 15526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:17.920496 15526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.920543 15526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:17.920584 15526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:17.926793 15526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46139
I20260928 17:54:17.926873 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46139 every 8 connection(s)
I20260928 17:54:17.927258 15526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:54:17.928455 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.928005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:51461)
0928 17:54:17.928220 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.928296 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
W20260928 17:54:17.928932 15655 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45315 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45315: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
I20260928 17:54:17.929235 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.929141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:41407)
0928 17:54:17.929187 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.929216 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:54:17.929499 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:17.929438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:47837)
0928 17:54:17.929461 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:17.929484 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 17:54:17.929631 15655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:54:17.936973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15526
I20260928 17:54:18.760620 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.760359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:52007)
0928 17:54:18.760518 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:18.760558 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260928 17:54:18.810997 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.810773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:44981)
0928 17:54:18.810912 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:18.810948 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260928 17:54:18.888434 15661 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:18.888559 15661 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:18.888862 15661 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139), b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
Sep 28 17:54:18 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 17:54:18 dist-test-slave-zqlt krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 17:54:18.899436 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.889160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48799 (local address 127.0.98.2:46235)
0928 17:54:18.889316 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:18.889324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:18.889349 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:54:18.889755 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:18.889757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:18.889795 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:18.889940 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:18.890635 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.891518 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.892644 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.893030 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.893154 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:18.895807 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:18.895828 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:18.895829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:18.895840 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:18.897670 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898102 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:18.898104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:18.898105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:18.898194 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898506 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:18.898507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:18.898507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:18.898797 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:18.898884 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:18.899247 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 17:54:18.899363 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260928 17:54:18.899442 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.889003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46235 (local address 127.0.98.1:48799)
0928 17:54:18.889128 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:18.889141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:18.889196 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:18.889988 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:18.889990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:18.890345 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:18.890522 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:18.890527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.891616 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.891617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:18.892525 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:18.892527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.893124 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.893125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:18.893225 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:18.893797 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:18.893806 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:18.895717 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:18.897747 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:18.897748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:18.897748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:18.898026 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:18.898271 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:18.898272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:18.898433 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:18.898879 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:18.898886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:18.899038 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 17:54:18.899167 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 17:54:18.899338 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":87,"thread_start_us":51,"threads_started":1}
I20260928 17:54:18.899621 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.889119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35181 (local address 127.0.98.3:46139)
0928 17:54:18.889262 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:18.889275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:18.889301 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:54:18.889744 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:18.889746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:18.889796 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:18.889939 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:18.890637 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.891547 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.892605 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.893002 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.893138 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:18.895365 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:18.895368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:18.895369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:18.895377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:18.897433 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898084 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:18.898087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:18.898088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:18.898194 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898534 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:18.898535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:18.898535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:18.898797 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:18.898884 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:18.899341 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 17:54:18.899470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":48,"threads_started":1}
I20260928 17:54:18.899742 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.888990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46139 (local address 127.0.98.1:35181)
0928 17:54:18.889042 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:18.889056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:18.889120 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:18.889987 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:18.889990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:18.890344 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:18.890522 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:18.890527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.891640 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.891641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:18.892524 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:18.892526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:18.893111 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:18.893113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:18.893224 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:18.893852 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:18.893864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:18.895266 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:18.897556 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:18.897563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:18.897565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:18.897979 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:18.898271 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:18.898272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:18.898272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:18.898433 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:18.898879 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:18.898886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:18.899033 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 17:54:18.899175 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 17:54:18.899656 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":344}
W20260928 17:54:18.900563 15277 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
W20260928 17:54:18.900682 15278 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:18.900743 15278 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9; no voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969
I20260928 17:54:18.900884 15661 raft_consensus.cc:2726] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:54:18.930159 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:18.929934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:33785)
0928 17:54:18.930083 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:18.930125 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260928 17:54:19.761428 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:19.761151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:36291)
0928 17:54:19.761317 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:19.761371 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260928 17:54:19.811683 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:19.811479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:50159)
0928 17:54:19.811607 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:19.811644 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 17:54:19.930934 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:19.930685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:54325)
0928 17:54:19.930831 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:19.930877 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260928 17:54:20.762219 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:20.761963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:33741)
0928 17:54:20.762127 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:20.762173 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 17:54:20.812508 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:20.812266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:50327)
0928 17:54:20.812416 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:20.812462 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 17:54:20.835842 15671 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:20.835948 15671 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:20.836158 15671 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139), b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
W20260928 17:54:20.836576 15278 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
W20260928 17:54:20.836665 15277 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:20.836761 15277 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9; no voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969
I20260928 17:54:20.836861 15671 raft_consensus.cc:2726] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:54:20.931810 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:20.931544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:38167)
0928 17:54:20.931695 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:20.931738 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260928 17:54:21.763062 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:21.762826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:39017)
0928 17:54:21.762960 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:21.763014 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260928 17:54:21.813277 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:21.813054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:36793)
0928 17:54:21.813195 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:21.813234 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260928 17:54:21.932662 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:21.932376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:38269)
0928 17:54:21.932517 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:21.932591 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260928 17:54:22.763921 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:22.763667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:35423)
0928 17:54:22.763827 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:22.763873 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260928 17:54:22.813977 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:22.813763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:55469)
0928 17:54:22.813887 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:22.813928 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 17:54:22.933411 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:22.933203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:39911)
0928 17:54:22.933334 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:22.933371 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 17:54:23.441715 15679 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:23.441803 15679 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:23.442052 15679 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139), b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
W20260928 17:54:23.442408 15278 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
W20260928 17:54:23.442471 15277 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:23.442520 15277 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9; no voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969
I20260928 17:54:23.442618 15679 raft_consensus.cc:2726] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:54:23.764678 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:23.764454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:38733)
0928 17:54:23.764586 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:23.764626 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260928 17:54:23.814659 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:23.814446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:58381)
0928 17:54:23.814578 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:23.814615 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 17:54:23.934217 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:23.934025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:36339)
0928 17:54:23.934144 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:23.934180 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 17:54:24.765471 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:24.765228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:38071)
0928 17:54:24.765375 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:24.765421 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 17:54:24.815424 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:24.815191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:37765)
0928 17:54:24.815333 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:24.815377 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 17:54:24.935022 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:24.934748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:38755)
0928 17:54:24.934926 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:24.934974 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260928 17:54:25.766358 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:25.766109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:56207)
0928 17:54:25.766247 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:25.766309 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 17:54:25.816275 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:25.815954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:59123)
0928 17:54:25.816179 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:25.816227 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260928 17:54:25.935731 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:25.935526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:39279)
0928 17:54:25.935651 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:25.935687 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 17:54:26.767118 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:26.766895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:56147)
0928 17:54:26.767029 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:26.767069 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260928 17:54:26.816967 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:26.816777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:53821)
0928 17:54:26.816891 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:26.816921 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260928 17:54:26.936496 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:26.936232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:41187)
0928 17:54:26.936394 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:26.936448 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260928 17:54:27.767884 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:27.767652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:37835)
0928 17:54:27.767793 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:27.767837 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 17:54:27.817682 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:27.817490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:52191)
0928 17:54:27.817604 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:27.817639 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260928 17:54:27.937347 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:27.937070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:50601)
0928 17:54:27.937243 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:27.937297 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260928 17:54:28.358539 15695 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:28.358680 15695 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:28.358908 15695 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139), b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
W20260928 17:54:28.359340 15277 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
W20260928 17:54:28.359491 15278 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:28.359557 15278 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9; no voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969
I20260928 17:54:28.359619 15695 raft_consensus.cc:2726] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:54:28.768560 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:28.768350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:35111)
0928 17:54:28.768477 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:28.768514 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260928 17:54:28.818501 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:28.818282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:34037)
0928 17:54:28.818417 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:28.818452 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 17:54:28.938092 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:28.937873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:39971)
0928 17:54:28.938011 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:28.938046 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260928 17:54:29.769203 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:29.768994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:49631)
0928 17:54:29.769119 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:29.769158 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 17:54:29.819347 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:29.819059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:49167)
0928 17:54:29.819239 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:29.819289 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260928 17:54:29.938928 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:29.938668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:43917)
0928 17:54:29.938826 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:29.938870 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260928 17:54:30.769850 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:30.769619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:52087)
0928 17:54:30.769765 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:30.769803 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260928 17:54:30.820076 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:30.819846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:47699)
0928 17:54:30.819987 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:30.820030 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 17:54:30.939743 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:30.939541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:38145)
0928 17:54:30.939663 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:30.939697 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 17:54:31.122908 15705 raft_consensus.cc:482] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:31.123008 15705 raft_consensus.cc:504] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc336961489428bb43f8f0c74c19969" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46139 } } peers { permanent_uuid: "b3b9bbb2a1ca4254893a25fb1c167371" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46235 } } peers { permanent_uuid: "4369a573873441dca24fb6bbad585ed9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40817 } }
I20260928 17:54:31.123235 15705 leader_election.cc:288] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139), b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235)
W20260928 17:54:31.123631 15278 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dcc336961489428bb43f8f0c74c19969 (127.0.98.3:46139): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
W20260928 17:54:31.123741 15277 leader_election.cc:341] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b9bbb2a1ca4254893a25fb1c167371 (127.0.98.2:46235): Not found: Tablet not found: cd777fbd075d479185233d45ebc2b75b
I20260928 17:54:31.123818 15277 leader_election.cc:302] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4369a573873441dca24fb6bbad585ed9; no voters: b3b9bbb2a1ca4254893a25fb1c167371, dcc336961489428bb43f8f0c74c19969
I20260928 17:54:31.123910 15705 raft_consensus.cc:2726] T cd777fbd075d479185233d45ebc2b75b P 4369a573873441dca24fb6bbad585ed9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:54:31.770663 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:31.770424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:49263)
0928 17:54:31.770566 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:31.770616 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260928 17:54:31.820935 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:31.820675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:35681)
0928 17:54:31.820839 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:31.820889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 17:54:31.940475 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:31.940246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:50323)
0928 17:54:31.940397 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:31.940429 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260928 17:54:32.771544 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:32.771298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.1:33549)
0928 17:54:32.771460 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:32.771500 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260928 17:54:32.821666 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:32.821457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.2:56695)
0928 17:54:32.821574 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:32.821615 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260928 17:54:32.940645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15230
I20260928 17:54:32.941959 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:32.941362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45315 (local address 127.0.98.3:39759)
0928 17:54:32.941846 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:32.941893 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":59,"threads_started":1}
I20260928 17:54:32.945470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15393
I20260928 17:54:32.950238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15526
2026-09-28T17:54:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18184 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:54:33 dist-test-slave-zqlt krb5kdc[15714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:54:33 dist-test-slave-zqlt krb5kdc[15714](info): set up 2 sockets
Sep 28 17:54:33 dist-test-slave-zqlt krb5kdc[15714](info): commencing operation
krb5kdc: starting...
W20260928 17:54:33.992417 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:54:34.111380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:34.191829 15727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:34.192071 15727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:34.192104 15727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:34.193251 15727 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:54:34.193317 15727 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:54:34.193339 15727 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:54:34.193357 15727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:34.193389 15727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:34.193415 15727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:34.193452 15727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:34.193483 15727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:34.194947 15727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:34.195214 15727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:34.195429 15727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:34.200716 15735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:34.200747 15733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.200784 15727 server_base.cc:1034] running on GCE node
W20260928 17:54:34.200727 15732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.201152 15727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:34.201480 15727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:34.202641 15727 hybrid_clock.cc:648] HybridClock initialized: now 1790618074202641 us; error 41 us; skew 500 ppm
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:34.205039 15727 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:34.205497 15727 webserver.cc:460] Webserver started at http://127.0.98.62:34461/ using document root <none> and password file <none>
I20260928 17:54:34.205730 15727 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:34.205796 15727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:34.205951 15727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:34.206933 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "0d1df16c213041ec9e75fe01e23a9be7"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "64713f98dff42f751cadfae68dc7165a"
server_key_iv: "df80b6db41f984dc97e0f2bd374ed671"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.207291 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d1df16c213041ec9e75fe01e23a9be7"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "64713f98dff42f751cadfae68dc7165a"
server_key_iv: "df80b6db41f984dc97e0f2bd374ed671"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.208652 15727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:54:34.209347 15742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.209522 15727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.209625 15727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "0d1df16c213041ec9e75fe01e23a9be7"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "64713f98dff42f751cadfae68dc7165a"
server_key_iv: "df80b6db41f984dc97e0f2bd374ed671"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.209698 15727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:34.220338 15727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:34.222213 15727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:34.222381 15727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:34.226449 15727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39679
I20260928 17:54:34.226502 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39679 every 8 connection(s)
I20260928 17:54:34.226864 15727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:34.227865 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:34.230273 15795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap starting.
I20260928 17:54:34.230896 15795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:34.231207 15795 log.cc:824] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:34.231894 15795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: No bootstrap required, opened a new log
I20260928 17:54:34.233533 15795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:34.233669 15795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:34.233692 15795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Initialized, Role: FOLLOWER
I20260928 17:54:34.233868 15795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:34.233934 15795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:34.233956 15795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:34.234027 15795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:34.234551 15795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:34.234665 15795 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1df16c213041ec9e75fe01e23a9be7; no voters:
I20260928 17:54:34.234854 15795 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:54:34.234917 15798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:34.235097 15798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 LEADER]: Becoming Leader. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Running, Role: LEADER
I20260928 17:54:34.235112 15795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:34.235193 15798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:34.235590 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:54:34.235755 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:34.235611 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1df16c213041ec9e75fe01e23a9be7. Latest consensus state: current_term: 1 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:54:34.236007 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:34.236135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15727
I20260928 17:54:34.236248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:54:34.236250 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:34.236507 392 external_mini_cluster.cc:1428] Setting key 4e5b15b2f5de055f3687d0cca7ed3c70
I20260928 17:54:34.236937 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:34.238690 15805 catalog_manager.cc:1347] Generated new cluster ID: 16d0b983882f4574a2915067c635c01b
I20260928 17:54:34.238742 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:34.246790 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.237213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.0.98.62:39679)
0928 17:54:34.237592 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.237602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.237631 (+ 29us) server_negotiation.cc:407] Connection header received
0928 17:54:34.237744 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.237750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.237803 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.237938 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:34.238748 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.239642 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.240528 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.240991 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.241127 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.243454 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:34.243457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:34.243458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:34.243476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:34.245218 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.245639 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.245643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.245643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.245739 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.245961 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.245962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.245962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.246189 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:34.246340 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.246456 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.246597 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":72,"threads_started":1}
I20260928 17:54:34.254015 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:54:34.254556 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:34.262897 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Generated new TSK 0
I20260928 17:54:34.263087 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:54:34.290796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:34.373009 15821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:34.373227 15821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:34.373251 15821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:34.374352 15821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:34.374389 15821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:34.374415 15821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:34.375897 15821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:34.376127 15821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:34.376335 15821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:34.381275 15829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:34.381271 15827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.381500 15821 server_base.cc:1034] running on GCE node
W20260928 17:54:34.381268 15826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.381749 15821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:34.382032 15821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:34.383194 15821 hybrid_clock.cc:648] HybridClock initialized: now 1790618074383183 us; error 27 us; skew 500 ppm
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:34.385761 15821 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:34.386204 15821 webserver.cc:460] Webserver started at http://127.0.98.1:34753/ using document root <none> and password file <none>
I20260928 17:54:34.386387 15821 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:34.386426 15821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:34.386512 15821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:34.387509 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e36336cb5e94aab9fd48dd53a31c500"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "776d1ad24f3b1e2b858fbb5dc7f154fd"
server_key_iv: "437a690fd8271a1d3c9cfef3b3e1a14e"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.387802 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e36336cb5e94aab9fd48dd53a31c500"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "776d1ad24f3b1e2b858fbb5dc7f154fd"
server_key_iv: "437a690fd8271a1d3c9cfef3b3e1a14e"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.389082 15821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:54:34.389724 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.389891 15821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:54:34.389966 15821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "3e36336cb5e94aab9fd48dd53a31c500"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "776d1ad24f3b1e2b858fbb5dc7f154fd"
server_key_iv: "437a690fd8271a1d3c9cfef3b3e1a14e"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.390015 15821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:34.405313 15821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:34.407586 15821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:34.407747 15821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:34.407974 15821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:34.408301 15821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:34.408334 15821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.408355 15821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:34.408401 15821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.414232 15821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44061
I20260928 17:54:34.414279 15949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44061 every 8 connection(s)
I20260928 17:54:34.414578 15821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:34.415665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15821
I20260928 17:54:34.415783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:54:34.415975 392 external_mini_cluster.cc:1428] Setting key 5d4730f865113401afa59177eddb7ed7
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 17:54:34.423995 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.415218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55399 (local address 127.0.98.62:39679)
0928 17:54:34.415258 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.415260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.415469 (+ 209us) server_negotiation.cc:407] Connection header received
0928 17:54:34.416009 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.416010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.416027 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.416116 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:34.417034 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.417570 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.418578 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.418810 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.418898 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.421163 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:34.421164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:34.421165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:34.421175 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:34.422596 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.423009 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.423010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.423011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.423069 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.423351 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.423352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.423353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.423480 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:34.423558 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.423822 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.423905 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:54:34.424346 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.415215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:55399)
0928 17:54:34.415354 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.415387 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.415902 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.416187 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.416190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.416746 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:34.416926 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.416931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.417678 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.417679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.418486 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.418488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.418908 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.418909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.419065 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.419648 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:34.419657 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:34.421068 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:34.422696 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.422703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.422705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.422927 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.423142 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.423143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.423143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.423249 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.423549 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:34.423556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:34.423691 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.423800 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.423959 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260928 17:54:34.424868 15950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:34.424976 15950 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:34.425206 15950 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 17:54:34.425786 15759 ts_manager.cc:194] Registered new tserver with Master: 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:34.426695 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:54:34.439111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:34.521241 15955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:34.521488 15955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:34.521529 15955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:34.522681 15955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:34.522884 15955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:34.522989 15955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:34.524600 15955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:34.524868 15955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:34.525118 15955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:34.530231 15961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.530303 15955 server_base.cc:1034] running on GCE node
W20260928 17:54:34.530311 15960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:34.530231 15963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.530618 15955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:34.530861 15955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:34.532033 15955 hybrid_clock.cc:648] HybridClock initialized: now 1790618074532033 us; error 47 us; skew 500 ppm
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:34.534581 15955 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:34.534962 15955 webserver.cc:460] Webserver started at http://127.0.98.2:43295/ using document root <none> and password file <none>
I20260928 17:54:34.535199 15955 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:34.535245 15955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:34.535338 15955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:34.536481 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "732d939282104f8a96b32b50bcfa6309"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "33ba2c7877fe93b43950a0d1b4bfb4a1"
server_key_iv: "31d1a30f480901626d6b3204e526eb9b"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.536788 15955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "732d939282104f8a96b32b50bcfa6309"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "33ba2c7877fe93b43950a0d1b4bfb4a1"
server_key_iv: "31d1a30f480901626d6b3204e526eb9b"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.538092 15955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:34.538822 15970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.539016 15955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:34.539120 15955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "732d939282104f8a96b32b50bcfa6309"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "33ba2c7877fe93b43950a0d1b4bfb4a1"
server_key_iv: "31d1a30f480901626d6b3204e526eb9b"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.539193 15955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:34.550166 15955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:34.552052 15955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:34.552264 15955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:34.552524 15955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:34.552847 15955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:34.552881 15955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.552903 15955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:34.552958 15955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.558660 15955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36177
I20260928 17:54:34.558743 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36177 every 8 connection(s)
I20260928 17:54:34.559093 15955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:54:34.564252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 15955
I20260928 17:54:34.564343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:54:34.564551 392 external_mini_cluster.cc:1428] Setting key 199006525dd4b99e137a8afb9e959e8b
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:34.569247 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.559864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49981 (local address 127.0.98.62:39679)
0928 17:54:34.559963 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.559966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.560166 (+ 200us) server_negotiation.cc:407] Connection header received
0928 17:54:34.560721 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.560723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.560736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.560780 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:34.561761 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.562266 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.563287 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.563488 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.563578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.566233 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:34.566234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:34.566235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:34.566242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:34.567704 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.568241 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.568242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.568242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.568297 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.568584 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.568584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.568584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.568689 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:34.568738 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.569109 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.569182 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 17:54:34.569593 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.559816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:49981)
0928 17:54:34.560008 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.560045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.560606 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.560869 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.560872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.561459 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:34.561655 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.561660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.562376 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.562379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.563194 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.563196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.563586 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.563587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.563719 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.564298 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:34.564327 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:34.566131 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:34.567790 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.567796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.567798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.568171 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.568367 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.568368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.568368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.568513 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.568755 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:34.568761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:34.568951 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.569070 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.569264 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
I20260928 17:54:34.570173 16084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:34.570286 16084 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:34.570500 16084 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260928 17:54:34.570851 15759 ts_manager.cc:194] Registered new tserver with Master: 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
I20260928 17:54:34.571237 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49981
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:54:34.590445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:34.670325 16089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:34.670581 16089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:34.670620 16089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:34.671756 16089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:34.671825 16089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:34.671868 16089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:34.673346 16089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:34.673611 16089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:34.673835 16089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:34.678896 16095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.678951 16089 server_base.cc:1034] running on GCE node
W20260928 17:54:34.678951 16094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:34.678912 16097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:34.679301 16089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:34.679548 16089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:34.680739 16089 hybrid_clock.cc:648] HybridClock initialized: now 1790618074680738 us; error 37 us; skew 500 ppm
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:34.683372 16089 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:34.683763 16089 webserver.cc:460] Webserver started at http://127.0.98.3:44565/ using document root <none> and password file <none>
I20260928 17:54:34.683954 16089 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:34.683993 16089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:34.684134 16089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:54:34.685258 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "6383d43b195d437a84b5cbe67f2f9120"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "7889f5f0315dedbce388b0d5d33da7f8"
server_key_iv: "ba23c41cd3e603b886e9ebe2aed06a1c"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.685571 16089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6383d43b195d437a84b5cbe67f2f9120"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "7889f5f0315dedbce388b0d5d33da7f8"
server_key_iv: "ba23c41cd3e603b886e9ebe2aed06a1c"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.686897 16089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:34.687628 16104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.687801 16089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:34.687885 16089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "6383d43b195d437a84b5cbe67f2f9120"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "7889f5f0315dedbce388b0d5d33da7f8"
server_key_iv: "ba23c41cd3e603b886e9ebe2aed06a1c"
server_key_version: "encryptionkey@0"
I20260928 17:54:34.687954 16089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:34.698802 16089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:34.701082 16089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:34.701188 16089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:34.701359 16089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:34.701607 16089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:34.701637 16089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.701658 16089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:34.701670 16089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:34.707255 16089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34939
I20260928 17:54:34.707286 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34939 every 8 connection(s)
I20260928 17:54:34.707615 16089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:54:34 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:34.715034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16089
I20260928 17:54:34.715154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:54:34.715358 392 external_mini_cluster.cc:1428] Setting key 52a3dfda1b77c796c9a29afff9178dd2
I20260928 17:54:34.716921 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.708370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40529 (local address 127.0.98.62:39679)
0928 17:54:34.708437 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.708439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.708645 (+ 206us) server_negotiation.cc:407] Connection header received
0928 17:54:34.709113 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.709114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.709124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.709175 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:34.710064 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.710557 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.711493 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.711675 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.711727 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.713939 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:34.713941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:34.713941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:34.713948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:34.715421 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.715908 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.715909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.715910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.715974 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.716267 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.716268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.716268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.716394 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:34.716476 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.716740 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.716830 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:54:34.717121 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.708377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:40529)
0928 17:54:34.708528 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.708563 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.709029 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.709232 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.709235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.709795 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:34.709983 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.709987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.710636 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.710636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.711423 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.711425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.711763 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.711763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.711883 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.712363 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:34.712371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:34.713852 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:34.715513 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.715519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.715521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.715808 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.716045 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:34.716046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:34.716046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:34.716193 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:34.716464 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:34.716471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:34.716610 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.716718 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.716883 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260928 17:54:34.717564 16218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:34.717643 16218 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:34.717837 16218 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260928 17:54:34.718209 15759 ts_manager.cc:194] Registered new tserver with Master: 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939)
I20260928 17:54:34.718596 15759 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40529
I20260928 17:54:34.727255 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:34.735332 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.728821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.62:39679)
0928 17:54:34.728872 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.728874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.729014 (+ 140us) server_negotiation.cc:407] Connection header received
0928 17:54:34.729089 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.729090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.729099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.729152 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:34.729793 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.730277 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.731049 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.731225 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.731275 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.732761 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:34.732762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:34.732763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:34.732769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:34.734176 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.734508 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.734508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.734509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.734557 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:34.734789 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:34.734789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:34.734790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:34.734950 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:34.735024 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.735167 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.735227 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:54:34.737486 15759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:54:34.739166 15759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:54:34.745831 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.0.1:45412)
0928 17:54:34.741765 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.741773 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.741835 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.742446 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.742447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.742449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:34.742492 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.742494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743511 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.743512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.744554 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.744557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745534 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.745610 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.745623 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.745696 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.745752 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":75,"thread_start_us":64,"threads_started":1}
I20260928 17:54:34.745966 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36177 (local address 127.0.0.1:43832)
0928 17:54:34.741687 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.741725 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.741835 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.742157 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.742162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.742164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:34.742242 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.742245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743580 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.743581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.744866 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.744869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745660 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.745731 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.745734 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.745778 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.745811 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":43,"thread_start_us":55,"threads_started":1}
I20260928 17:54:34.745993 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.98.2:36177)
0928 17:54:34.741968 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.741974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.741986 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:54:34.742030 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.742032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.742049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.742138 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:34.742336 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743485 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.744955 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745505 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745573 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.745629 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.745869 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.745913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260928 17:54:34.745879 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.0.98.1:44061)
0928 17:54:34.742270 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.742274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.742284 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:34.742320 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.742323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.742336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.742565 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:34.742629 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743405 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.744651 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745441 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745521 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.745578 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.745727 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.745771 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":83,"threads_started":1}
I20260928 17:54:34.746105 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34939 (local address 127.0.0.1:43318)
0928 17:54:34.741690 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:34.741936 (+ 246us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:34.741975 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:34.742334 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:34.742335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:34.742336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:34.742422 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.742424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743711 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.743711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.744963 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:34.744965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745884 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:34.745953 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.745956 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:54:34.745988 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 17:54:34.746030 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":124,"thread_start_us":54,"threads_started":1}
I20260928 17:54:34.746747 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:34.741962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.0.98.3:34939)
0928 17:54:34.742162 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:34.742167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:34.742176 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:34.742211 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:34.742215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:34.742229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:34.742496 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:34.742568 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.743648 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.745039 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:34.745770 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:34.746553 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:34.746616 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:34.746640 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 17:54:34.746688 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260928 17:54:34.746618 15884 tablet_service.cc:1467] Processing CreateTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 (DEFAULT_TABLE table=test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:34.747030 15884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776e44a7f8d340cdb028d186e1a4ca28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:34.747044 16018 tablet_service.cc:1467] Processing CreateTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 (DEFAULT_TABLE table=test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:34.747337 16018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776e44a7f8d340cdb028d186e1a4ca28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:34.747377 16152 tablet_service.cc:1467] Processing CreateTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 (DEFAULT_TABLE table=test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:34.747720 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776e44a7f8d340cdb028d186e1a4ca28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:34.750499 16233 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Bootstrap starting.
I20260928 17:54:34.750732 16232 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap starting.
I20260928 17:54:34.750808 16234 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Bootstrap starting.
I20260928 17:54:34.751350 16234 tablet_bootstrap.cc:654] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:34.751358 16232 tablet_bootstrap.cc:654] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:34.751847 16234 log.cc:824] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:34.751875 16232 log.cc:824] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:34.752593 16234 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: No bootstrap required, opened a new log
I20260928 17:54:34.752637 16233 tablet_bootstrap.cc:654] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:34.752677 16234 ts_tablet_manager.cc:1397] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:34.753095 16233 log.cc:824] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:34.753782 16233 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: No bootstrap required, opened a new log
I20260928 17:54:34.753870 16233 ts_tablet_manager.cc:1397] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 17:54:34.754297 16234 raft_consensus.cc:348] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.754443 16234 raft_consensus.cc:374] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:34.754467 16234 raft_consensus.cc:729] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6383d43b195d437a84b5cbe67f2f9120, State: Initialized, Role: FOLLOWER
I20260928 17:54:34.754858 16232 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: No bootstrap required, opened a new log
I20260928 17:54:34.754914 16232 ts_tablet_manager.cc:1397] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 17:54:34.755412 16233 raft_consensus.cc:348] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.755573 16233 raft_consensus.cc:374] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:34.755632 16233 raft_consensus.cc:729] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732d939282104f8a96b32b50bcfa6309, State: Initialized, Role: FOLLOWER
I20260928 17:54:34.755801 16233 consensus_queue.cc:260] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.756145 16233 ts_tablet_manager.cc:1428] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:54:34.756176 16084 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260928 17:54:34.756462 16234 consensus_queue.cc:260] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.756680 16232 raft_consensus.cc:348] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.756819 16234 ts_tablet_manager.cc:1428] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 17:54:34.756851 16218 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260928 17:54:34.756934 16232 raft_consensus.cc:374] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:34.756989 16232 raft_consensus.cc:729] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e36336cb5e94aab9fd48dd53a31c500, State: Initialized, Role: FOLLOWER
I20260928 17:54:34.757107 16232 consensus_queue.cc:260] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:34.757395 16232 ts_tablet_manager.cc:1428] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:54:34.757432 15950 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
W20260928 17:54:34.809679 16085 tablet.cc:2369] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:54:34.915313 15951 tablet.cc:2369] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:54:34.958209 16219 tablet.cc:2369] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:54:35.099853 16238 raft_consensus.cc:482] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:35.100040 16238 raft_consensus.cc:504] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:35.100500 16238 leader_election.cc:288] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:35.103408 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.100820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41071 (local address 127.0.98.1:44061)
0928 17:54:35.100875 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:35.100877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:35.100921 (+ 44us) server_negotiation.cc:407] Connection header received
0928 17:54:35.101025 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:35.101028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:35.101043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:35.101114 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:35.101268 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.101747 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.102711 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.103067 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.103120 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:35.103128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:35.103297 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 17:54:35.103342 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:54:35.103421 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.100726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.98.2:41071)
0928 17:54:35.100859 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:35.100869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:35.100936 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:35.101119 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:35.101120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:35.101122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:35.101200 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:35.101202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.101827 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.101828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:35.102611 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:35.102613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.103144 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.103145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:35.103217 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:35.103221 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:35.103275 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:54:35.103305 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260928 17:54:35.103482 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.100781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45137 (local address 127.0.98.3:34939)
0928 17:54:35.100849 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:35.100851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:35.100908 (+ 57us) server_negotiation.cc:407] Connection header received
0928 17:54:35.101021 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:35.101022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:35.101034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:35.101094 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:35.101299 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.101767 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.102712 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.103117 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.103185 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:35.103193 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:35.103364 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 17:54:35.103404 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:54:35.103698 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.100688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34939 (local address 127.0.98.2:45137)
0928 17:54:35.100835 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:35.100846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:35.100926 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:35.101113 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:35.101114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:35.101116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:35.101211 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:35.101213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.101883 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.101884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:35.102613 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:35.102615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.103211 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.103212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:35.103285 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:35.103290 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:54:35.103347 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 17:54:35.103384 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260928 17:54:35.103956 15904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e44a7f8d340cdb028d186e1a4ca28" candidate_uuid: "732d939282104f8a96b32b50bcfa6309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e36336cb5e94aab9fd48dd53a31c500" is_pre_election: true
I20260928 17:54:35.104131 15904 raft_consensus.cc:2454] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732d939282104f8a96b32b50bcfa6309 in term 0.
I20260928 17:54:35.104120 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e44a7f8d340cdb028d186e1a4ca28" candidate_uuid: "732d939282104f8a96b32b50bcfa6309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6383d43b195d437a84b5cbe67f2f9120" is_pre_election: true
I20260928 17:54:35.104276 16172 raft_consensus.cc:2454] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732d939282104f8a96b32b50bcfa6309 in term 0.
I20260928 17:54:35.104382 15973 leader_election.cc:302] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500, 732d939282104f8a96b32b50bcfa6309; no voters:
I20260928 17:54:35.104522 16238 raft_consensus.cc:2781] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:35.104584 16238 raft_consensus.cc:482] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:35.104601 16238 raft_consensus.cc:3037] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:35.105309 16238 raft_consensus.cc:504] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:35.105510 16238 leader_election.cc:288] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [CANDIDATE]: Term 1 election: Requested vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:35.105602 15904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e44a7f8d340cdb028d186e1a4ca28" candidate_uuid: "732d939282104f8a96b32b50bcfa6309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e36336cb5e94aab9fd48dd53a31c500"
I20260928 17:54:35.105679 15904 raft_consensus.cc:3037] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:35.106346 15904 raft_consensus.cc:2454] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732d939282104f8a96b32b50bcfa6309 in term 1.
I20260928 17:54:35.106484 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776e44a7f8d340cdb028d186e1a4ca28" candidate_uuid: "732d939282104f8a96b32b50bcfa6309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6383d43b195d437a84b5cbe67f2f9120"
I20260928 17:54:35.106530 15973 leader_election.cc:302] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500, 732d939282104f8a96b32b50bcfa6309; no voters:
I20260928 17:54:35.106565 16172 raft_consensus.cc:3037] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:35.106609 16238 raft_consensus.cc:2781] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:35.106782 16238 raft_consensus.cc:686] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 LEADER]: Becoming Leader. State: Replica: 732d939282104f8a96b32b50bcfa6309, State: Running, Role: LEADER
I20260928 17:54:35.106881 16238 consensus_queue.cc:237] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:35.107424 16172 raft_consensus.cc:2454] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732d939282104f8a96b32b50bcfa6309 in term 1.
I20260928 17:54:35.107988 15758 catalog_manager.cc:5495] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732d939282104f8a96b32b50bcfa6309 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "732d939282104f8a96b32b50bcfa6309" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20260928 17:54:35.186054 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.183473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.0.98.2:36177)
0928 17:54:35.183548 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:35.183551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:35.183561 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:35.183605 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:35.183606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:35.183636 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:35.183718 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:35.183860 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.184411 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.185222 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:35.185432 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:35.185511 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:35.185677 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:35.185813 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:35.185915 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:35.185936 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 17:54:35.185960 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:54:35.188024 15904 raft_consensus.cc:1261] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Refusing update from remote peer 732d939282104f8a96b32b50bcfa6309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:35.188345 16172 raft_consensus.cc:1261] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Refusing update from remote peer 732d939282104f8a96b32b50bcfa6309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:35.188504 16243 consensus_queue.cc:1035] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:35.188721 16238 consensus_queue.cc:1035] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:35.191033 16246 mvcc.cc:205] Tried to move back new op lower bound from 7334371635967152128 to 7334371635638599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334371635967152128}]
I20260928 17:54:35.191279 16248 mvcc.cc:205] Tried to move back new op lower bound from 7334371635967152128 to 7334371635638599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334371635967152128}]
I20260928 17:54:35.191931 16247 mvcc.cc:205] Tried to move back new op lower bound from 7334371635967152128 to 7334371635638599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334371635967152128}]
I20260928 17:54:35.193848 15758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:35.193950 15758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:35.194804 15758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Sending DeleteTablet for 3 replicas of tablet 776e44a7f8d340cdb028d186e1a4ca28
I20260928 17:54:35.195145 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43318
I20260928 17:54:35.195181 15884 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45412
I20260928 17:54:35.195345 16018 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43832
I20260928 17:54:35.195443 16256 tablet_replica.cc:330] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: stopping tablet replica
I20260928 17:54:35.195501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15821
I20260928 17:54:35.195534 16256 raft_consensus.cc:2229] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:35.195546 16255 tablet_replica.cc:330] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: stopping tablet replica
I20260928 17:54:35.195634 16255 raft_consensus.cc:2229] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:35.195652 16256 raft_consensus.cc:2258] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:35.195725 16255 raft_consensus.cc:2258] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:35.196636 16255 ts_tablet_manager.cc:1905] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:35.198366 16255 ts_tablet_manager.cc:1918] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:35.198483 16255 log.cc:1196] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/776e44a7f8d340cdb028d186e1a4ca28
I20260928 17:54:35.198774 16255 ts_tablet_manager.cc:1939] T 776e44a7f8d340cdb028d186e1a4ca28 P 6383d43b195d437a84b5cbe67f2f9120: Deleting consensus metadata
I20260928 17:54:35.199209 15743 catalog_manager.cc:4841] TS 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): tablet 776e44a7f8d340cdb028d186e1a4ca28 (table test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]) successfully deleted
W20260928 17:54:35.200984 15745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44061 (error 108)
I20260928 17:54:35.201233 16257 tablet_replica.cc:330] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: stopping tablet replica
I20260928 17:54:35.201350 16257 raft_consensus.cc:2229] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:54:35.201514 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.201384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.0.1:45426)
0928 17:54:35.201438 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:35.201484 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:54:35.201511 16257 raft_consensus.cc:2258] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 17:54:35.201644 15745 catalog_manager.cc:4584] TS 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 776e44a7f8d340cdb028d186e1a4ca28: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44061: connect: Connection refused (error 111)
I20260928 17:54:35.201697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15955
I20260928 17:54:35.206197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16089
I20260928 17:54:35.206274 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:35.206154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36177 (local address 127.0.0.1:43848)
0928 17:54:35.206204 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:35.206235 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:54:35.211509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 15727
I20260928 17:54:55.217242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--webserver_interface=127.0.98.62
--webserver_port=34461
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:54:55.296602 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:55.296809 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:55.296831 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:55.297993 16259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:54:55.298033 16259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:54:55.298046 16259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:54:55.298058 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:54:55.298079 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:55.298091 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:55.298103 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:54:55.298115 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:54:55.299537 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:55.299726 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:55.299888 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:55.309154 16266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.309135 16268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:55.309175 16259 server_base.cc:1034] running on GCE node
W20260928 17:54:55.309134 16265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:55.309656 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:55.309988 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:55.311161 16259 hybrid_clock.cc:648] HybridClock initialized: now 1790618095311144 us; error 43 us; skew 500 ppm
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:55.321291 16259 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:54:55.321844 16259 webserver.cc:460] Webserver started at http://127.0.98.62:34461/ using document root <none> and password file <none>
I20260928 17:54:55.322089 16259 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:55.322172 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:55.323951 16259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:54:55.324751 16275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.324963 16259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.325069 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "0d1df16c213041ec9e75fe01e23a9be7"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "64713f98dff42f751cadfae68dc7165a"
server_key_iv: "df80b6db41f984dc97e0f2bd374ed671"
server_key_version: "encryptionkey@0"
I20260928 17:54:55.325513 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:55.337348 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:55.339380 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:55.339545 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:55.344082 16259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39679
I20260928 17:54:55.344179 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39679 every 8 connection(s)
I20260928 17:54:55.344560 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:54:55.346133 16328 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:54:55.346765 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap starting.
I20260928 17:54:55.352342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16259
I20260928 17:54:55.352689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44061
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=34753
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:55.355504 16328 log.cc:824] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:55.357609 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:55.357925 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap complete.
I20260928 17:54:55.360280 16328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:55.360580 16328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Initialized, Role: FOLLOWER
I20260928 17:54:55.360743 16328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:55.360811 16328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:54:55.360874 16328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:54:55.360917 16328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 17:54:55.361922 16328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:55.362008 16328 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1df16c213041ec9e75fe01e23a9be7; no voters:
I20260928 17:54:55.362187 16328 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 17:54:55.362222 16332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 17:54:55.362332 16332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 LEADER]: Becoming Leader. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Running, Role: LEADER
I20260928 17:54:55.362448 16328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:54:55.362473 16332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:54:55.362885 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:54:55.362969 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:55.363180 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1df16c213041ec9e75fe01e23a9be7. Latest consensus state: current_term: 2 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:54:55.363278 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:54:55.363360 16338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:54:55.364172 16338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:54:55.364276 16338 catalog_manager.cc:1259] Loaded cluster ID: 16d0b983882f4574a2915067c635c01b
I20260928 17:54:55.364357 16338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:54:55.365556 16338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:54:55.365739 16338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Loaded TSK: 0
I20260928 17:54:55.375814 16338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Generated new TSK 1
I20260928 17:54:55.375910 16338 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 17:54:55.435902 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:55.436157 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:55.436219 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:55.437359 16330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:55.437430 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:55.437480 16330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:54:55.438916 16330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34753
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:55.439146 16330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:55.439388 16330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:55.445312 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.445345 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.445385 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:55.445771 16330 server_base.cc:1034] running on GCE node
I20260928 17:54:55.445972 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:55.446223 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:55.447384 16330 hybrid_clock.cc:648] HybridClock initialized: now 1790618095447365 us; error 34 us; skew 500 ppm
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:55.449918 16330 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:54:55.450415 16330 webserver.cc:460] Webserver started at http://127.0.98.1:34753/ using document root <none> and password file <none>
I20260928 17:54:55.450670 16330 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:55.450726 16330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:55.452139 16330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:55.452836 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.453013 16330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:54:55.453119 16330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "3e36336cb5e94aab9fd48dd53a31c500"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "776d1ad24f3b1e2b858fbb5dc7f154fd"
server_key_iv: "437a690fd8271a1d3c9cfef3b3e1a14e"
server_key_version: "encryptionkey@0"
I20260928 17:54:55.453550 16330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:55.468165 16330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:55.470551 16330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:55.470702 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:55.470969 16330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:55.471462 16372 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:54:55.472976 16330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:54:55.473034 16330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:54:55.473068 16330 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:54:55.473987 16330 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:54:55.474033 16330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.474108 16372 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap starting.
I20260928 17:54:55.481032 16330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44061
I20260928 17:54:55.481168 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44061 every 8 connection(s)
I20260928 17:54:55.481508 16330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:54:55.484346 16372 log.cc:824] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:55.487459 16372 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:55.487759 16372 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap complete.
I20260928 17:54:55.488191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16330
I20260928 17:54:55.488451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36177
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=43295
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:55.488982 16372 ts_tablet_manager.cc:1397] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260928 17:54:55.490705 16372 raft_consensus.cc:348] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.491226 16372 raft_consensus.cc:729] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e36336cb5e94aab9fd48dd53a31c500, State: Initialized, Role: FOLLOWER
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:55.491524 16372 consensus_queue.cc:260] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.491863 16372 ts_tablet_manager.cc:1428] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260928 17:54:55.496191 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.482719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33515 (local address 127.0.98.62:39679)
0928 17:54:55.484485 (+ 1766us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.484495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.484538 (+ 43us) server_negotiation.cc:407] Connection header received
0928 17:54:55.484661 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.484667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.484708 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.484864 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.485492 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.486881 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.488444 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.489021 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.489581 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.491903 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.491907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.491908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.491920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.494025 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.494567 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.494570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.494570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.494685 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.495017 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.495018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.495018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.495327 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.495424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.495813 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.495959 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":79,"threads_started":1}
I20260928 17:54:55.496203 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.482372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:33515)
0928 17:54:55.482589 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.482652 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.483384 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.484888 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.484891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.485185 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:55.485367 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.485371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.487032 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.487034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.488342 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.488344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.489155 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.489156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.489296 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.489934 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:55.489943 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:55.491770 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:55.494117 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.494123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.494126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.494467 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.494768 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.494768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.494769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.494934 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.495415 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:55.495422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:55.495596 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.495725 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.495918 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260928 17:54:55.497529 16480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:55.497630 16480 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:55.497820 16480 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260928 17:54:55.498618 16292 ts_manager.cc:194] Registered new tserver with Master: 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:55.499751 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33515
I20260928 17:54:55.500834 16480 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
I20260928 17:54:55.507412 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.500978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.0.98.1:44061)
0928 17:54:55.501151 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.501155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.503731 (+ 2576us) server_negotiation.cc:407] Connection header received
0928 17:54:55.503783 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.503785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.503800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.503888 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:55.504054 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.505086 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.506582 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.507027 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.507089 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.507123 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.507327 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.507361 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260928 17:54:55.507434 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.500158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.0.1:35142)
0928 17:54:55.503598 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.503636 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.503710 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.503906 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.503909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.503910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:55.503978 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.503980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.505185 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.505186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.506492 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.506494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.507114 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.507115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.507202 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.507215 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.507304 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.507357 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3409,"thread_start_us":81,"threads_started":1}
I20260928 17:54:55.507828 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35142
I20260928 17:54:55.508423 16489 tablet_replica.cc:330] stopping tablet replica
I20260928 17:54:55.508530 16489 raft_consensus.cc:2229] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:55.508596 16489 raft_consensus.cc:2258] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:55.508889 16489 ts_tablet_manager.cc:1905] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:55.511397 16489 ts_tablet_manager.cc:1918] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:55.511472 16489 log.cc:1196] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal/wals/776e44a7f8d340cdb028d186e1a4ca28
I20260928 17:54:55.511698 16489 ts_tablet_manager.cc:1939] T 776e44a7f8d340cdb028d186e1a4ca28 P 3e36336cb5e94aab9fd48dd53a31c500: Deleting consensus metadata
I20260928 17:54:55.512827 16278 catalog_manager.cc:4841] TS 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061): tablet 776e44a7f8d340cdb028d186e1a4ca28 (table test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]) successfully deleted
W20260928 17:54:55.579494 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:55.579694 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:55.579722 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:55.580930 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:55.580981 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:55.581017 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:54:55.582532 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43295
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:55.582718 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:55.582880 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:55.587882 16494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.587905 16495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.587890 16497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:55.588121 16485 server_base.cc:1034] running on GCE node
I20260928 17:54:55.588346 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:55.588584 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:55.589751 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790618095589745 us; error 37 us; skew 500 ppm
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:55.592396 16485 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:54:55.592864 16485 webserver.cc:460] Webserver started at http://127.0.98.2:43295/ using document root <none> and password file <none>
I20260928 17:54:55.593114 16485 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:55.593195 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:55.594514 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:54:55.595180 16504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.595352 16485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:54:55.595449 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "732d939282104f8a96b32b50bcfa6309"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "33ba2c7877fe93b43950a0d1b4bfb4a1"
server_key_iv: "31d1a30f480901626d6b3204e526eb9b"
server_key_version: "encryptionkey@0"
I20260928 17:54:55.595789 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:55.605541 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:55.607875 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:55.608045 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:55.608284 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:55.608701 16511 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:54:55.610108 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:54:55.610177 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:54:55.610221 16485 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:54:55.610873 16485 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:54:55.610929 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.611033 16511 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Bootstrap starting.
I20260928 17:54:55.618001 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36177
I20260928 17:54:55.618067 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36177 every 8 connection(s)
I20260928 17:54:55.618384 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:54:55.620913 16511 log.cc:824] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:55.623304 16511 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:54:55.623600 16511 tablet_bootstrap.cc:492] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Bootstrap complete.
I20260928 17:54:55.624759 16511 ts_tablet_manager.cc:1397] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:55.625571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16485
I20260928 17:54:55.625860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34939
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=44565
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:54:55.625960 16511 raft_consensus.cc:348] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.626330 16511 raft_consensus.cc:729] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732d939282104f8a96b32b50bcfa6309, State: Initialized, Role: FOLLOWER
I20260928 17:54:55.626497 16511 consensus_queue.cc:260] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.626791 16511 ts_tablet_manager.cc:1428] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 17:54:55.629161 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.619155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39979 (local address 127.0.98.62:39679)
0928 17:54:55.619228 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.619230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.619401 (+ 171us) server_negotiation.cc:407] Connection header received
0928 17:54:55.620142 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.620144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.620158 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.620230 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.621045 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.621573 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.623013 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.623202 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.623255 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.625756 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.625758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.625758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.625766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.627320 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.628031 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.628033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.628034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.628141 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.628426 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.628427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.628427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.628586 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.628660 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.628979 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.629070 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 17:54:55.629276 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.619075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:39979)
0928 17:54:55.619263 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.619304 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.620004 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.620297 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.620302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.620673 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:55.620944 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.620949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.621678 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.621679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.622928 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.622929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.623299 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.623300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.623456 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.624006 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:55.624015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:55.625666 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:55.627421 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.627427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.627430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.627915 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.628198 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.628198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.628199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.628326 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.628663 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:55.628670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:55.628804 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.628912 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.629078 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260928 17:54:55.629702 16619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:55.629810 16619 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:55.630003 16619 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260928 17:54:55.630559 16292 ts_manager.cc:194] Registered new tserver with Master: 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
I20260928 17:54:55.631067 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39979
I20260928 17:54:55.631942 16619 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 17:54:55.638811 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.630900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36177 (local address 127.0.0.1:58146)
0928 17:54:55.630960 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.630970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.631020 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.631401 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.631403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.631681 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:55.631712 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.631714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.632773 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.632774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.633359 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.633361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.634031 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.634034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.634103 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.634596 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:55.634600 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:55.635851 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:55.637652 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.637659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.637660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.637844 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.638154 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.638155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.638156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.638229 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.638545 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:55.638552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:55.638597 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.638664 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.638749 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:54:55.638895 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.631106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.98.2:36177)
0928 17:54:55.631258 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.631263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.631272 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:55.631308 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.631310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.631324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.631390 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.631804 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.632692 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.633623 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.633930 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.634019 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.635934 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.635942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.635943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.635950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.637565 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.637918 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.637920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.637921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.637985 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.638302 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.638302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.638303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.638460 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.638543 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.638714 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.638804 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260928 17:54:55.639271 16553 tablet_service.cc:1514] Processing DeleteTablet for tablet 776e44a7f8d340cdb028d186e1a4ca28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58146
I20260928 17:54:55.639810 16626 tablet_replica.cc:330] stopping tablet replica
I20260928 17:54:55.639940 16626 raft_consensus.cc:2229] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:55.640026 16626 raft_consensus.cc:2258] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:55.640370 16626 ts_tablet_manager.cc:1905] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:55.642450 16626 ts_tablet_manager.cc:1918] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:55.642549 16626 log.cc:1196] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/776e44a7f8d340cdb028d186e1a4ca28
I20260928 17:54:55.642836 16626 ts_tablet_manager.cc:1939] T 776e44a7f8d340cdb028d186e1a4ca28 P 732d939282104f8a96b32b50bcfa6309: Deleting consensus metadata
I20260928 17:54:55.643306 16277 catalog_manager.cc:4841] TS 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): tablet 776e44a7f8d340cdb028d186e1a4ca28 (table test-table [id=cff84ccbd3e841d0a7ebc5d57a38abf2]) successfully deleted
W20260928 17:54:55.710783 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:54:55.711007 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:54:55.711033 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:54:55.712234 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:54:55.712280 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:54:55.712308 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:54:55.713809 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44565
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:54:55.714076 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:54:55.714323 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:54:55.719358 16631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.719367 16634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:54:55.719476 16632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:54:55.719674 16624 server_base.cc:1034] running on GCE node
I20260928 17:54:55.719893 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:54:55.720162 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:54:55.721302 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790618095721290 us; error 32 us; skew 500 ppm
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:54:55.723748 16624 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:54:55.724200 16624 webserver.cc:460] Webserver started at http://127.0.98.3:44565/ using document root <none> and password file <none>
I20260928 17:54:55.724426 16624 fs_manager.cc:359] Metadata directory not provided
I20260928 17:54:55.724507 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:54:55.725854 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 17:54:55.726456 16641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.726632 16624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.726725 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "6383d43b195d437a84b5cbe67f2f9120"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "7889f5f0315dedbce388b0d5d33da7f8"
server_key_iv: "ba23c41cd3e603b886e9ebe2aed06a1c"
server_key_version: "encryptionkey@0"
I20260928 17:54:55.727072 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:54:55.735432 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:54:55.737994 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:54:55.738164 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:54:55.738404 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:54:55.738714 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:54:55.738768 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.738813 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:54:55.738845 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.745363 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34939
I20260928 17:54:55.745443 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34939 every 8 connection(s)
I20260928 17:54:55.745814 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:54:55.751566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16624
Sep 28 17:54:55 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:54:55.756615 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.746657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60445 (local address 127.0.98.62:39679)
0928 17:54:55.746752 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.746755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.746941 (+ 186us) server_negotiation.cc:407] Connection header received
0928 17:54:55.747517 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.747518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.747532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.747605 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.748606 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.749135 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.750157 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.750376 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.750470 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.753260 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.753262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.753262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.753270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.754916 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.755510 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.755511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.755512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.755576 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.755873 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.755873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.755874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.755973 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.756028 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.756401 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.756511 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 17:54:55.756769 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.746629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:60445)
0928 17:54:55.746816 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.746851 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.747363 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.747682 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.747685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.748302 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:54:55.748501 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.748505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.749239 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.749240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.750048 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.750050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.750477 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.750479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.750621 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.751207 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:54:55.751216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:54:55.753179 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:54:55.755017 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.755025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.755027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.755366 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.755650 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:54:55.755651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:54:55.755652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:54:55.755792 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:54:55.756043 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:54:55.756051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:54:55.756254 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.756388 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.756565 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260928 17:54:55.757465 16755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39679
I20260928 17:54:55.757582 16755 heartbeater.cc:464] Registering TS with master...
I20260928 17:54:55.757779 16755 heartbeater.cc:511] Master 127.0.98.62:39679 requested a full tablet report, sending...
I20260928 17:54:55.758189 16292 ts_manager.cc:194] Registered new tserver with Master: 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939)
I20260928 17:54:55.758577 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60445
I20260928 17:54:55.759399 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.752029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.0.98.62:39679)
0928 17:54:55.752155 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.752157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.752254 (+ 97us) server_negotiation.cc:407] Connection header received
0928 17:54:55.752322 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.752323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.752332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.752392 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.753190 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.753658 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.754571 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.754912 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.755007 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.756496 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.756498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.756498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.756504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.758207 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.758616 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.758617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.758618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.758699 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.758886 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.758887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.758887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.759002 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.759189 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.759226 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.759299 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":47,"threads_started":1}
I20260928 17:54:55.760036 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:54:55.762923 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.760436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.0.98.62:39679)
0928 17:54:55.760519 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.760521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.760614 (+ 93us) server_negotiation.cc:407] Connection header received
0928 17:54:55.760689 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.760689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.760729 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.760803 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:55.760901 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.761341 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.762089 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.762282 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.762353 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.762542 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:55.762589 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 17:54:55.762858 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51318: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 17:54:55.762986 16759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51318: token expired
W20260928 17:54:55.763089 16224 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39679)
I20260928 17:54:55.770466 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.763356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.98.62:39679)
0928 17:54:55.763419 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.763421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.763430 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:55.763452 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.763453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.763463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.763565 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:54:55.764314 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.764903 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.765997 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.766330 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.766412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.767534 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:54:55.767535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:54:55.767535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:54:55.767541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:54:55.769328 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.769680 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.769681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.769681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.769741 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:54:55.769971 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:54:55.769972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:54:55.769972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:54:55.770099 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:54:55.770195 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.770290 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.770362 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:54:55.771596 16292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:54:55.773224 16292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:54:55.775283 16553 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8705cd2cc44809b8d44be22e330baa (DEFAULT_TABLE table=test-table [id=453929e2cf934e8f9b663074e1af8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:55.775302 16414 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8705cd2cc44809b8d44be22e330baa (DEFAULT_TABLE table=test-table [id=453929e2cf934e8f9b663074e1af8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:55.775484 16553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8705cd2cc44809b8d44be22e330baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:55.775486 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8705cd2cc44809b8d44be22e330baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:55.776877 16511 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Bootstrap starting.
I20260928 17:54:55.777055 16372 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap starting.
I20260928 17:54:55.777206 16511 tablet_bootstrap.cc:654] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:55.777364 16372 tablet_bootstrap.cc:654] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:55.777710 16511 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: No bootstrap required, opened a new log
I20260928 17:54:55.777776 16511 ts_tablet_manager.cc:1397] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.777989 16372 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: No bootstrap required, opened a new log
I20260928 17:54:55.778026 16372 ts_tablet_manager.cc:1397] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.778116 16372 raft_consensus.cc:348] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.778148 16511 raft_consensus.cc:348] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.778173 16372 raft_consensus.cc:374] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:55.778203 16511 raft_consensus.cc:374] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:55.778215 16372 raft_consensus.cc:729] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e36336cb5e94aab9fd48dd53a31c500, State: Initialized, Role: FOLLOWER
I20260928 17:54:55.778230 16511 raft_consensus.cc:729] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732d939282104f8a96b32b50bcfa6309, State: Initialized, Role: FOLLOWER
I20260928 17:54:55.778275 16372 consensus_queue.cc:260] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.778295 16511 consensus_queue.cc:260] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.778388 16372 ts_tablet_manager.cc:1428] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 17:54:55.778398 16511 ts_tablet_manager.cc:1428] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 17:54:55.779618 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.775075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34939 (local address 127.0.0.1:59364)
0928 17:54:55.775160 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.775172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.775210 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.775668 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.775669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.775671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:55.775714 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.775715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.777113 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.777114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.778223 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.778226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.779444 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.779445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.779494 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.779497 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.779545 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.779574 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 17:54:55.779640 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.775300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.0.98.3:34939)
0928 17:54:55.775498 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.775503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.775513 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:54:55.775553 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.775555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.775570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.775657 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:55.775782 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.776990 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.778555 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.779385 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.779440 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.779478 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.779557 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.779585 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260928 17:54:55.780325 16689 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8705cd2cc44809b8d44be22e330baa (DEFAULT_TABLE table=test-table [id=453929e2cf934e8f9b663074e1af8a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:54:55.780659 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8705cd2cc44809b8d44be22e330baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:54:55.783228 16764 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Bootstrap starting.
I20260928 17:54:55.783879 16764 tablet_bootstrap.cc:654] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Neither blocks nor log segments found. Creating new log.
I20260928 17:54:55.784404 16764 log.cc:824] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Log is configured to *not* fsync() on all Append() calls
I20260928 17:54:55.784999 16764 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: No bootstrap required, opened a new log
I20260928 17:54:55.785053 16764 ts_tablet_manager.cc:1397] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:54:55.786304 16764 raft_consensus.cc:348] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.786440 16764 raft_consensus.cc:374] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:54:55.786463 16764 raft_consensus.cc:729] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6383d43b195d437a84b5cbe67f2f9120, State: Initialized, Role: FOLLOWER
I20260928 17:54:55.786573 16764 consensus_queue.cc:260] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.786798 16764 ts_tablet_manager.cc:1428] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:54:55.786854 16755 heartbeater.cc:503] Master 127.0.98.62:39679 was elected leader, sending a full tablet report...
W20260928 17:54:55.868975 16620 tablet.cc:2369] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:54:55.906404 16766 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:54:55.906567 16766 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.907091 16766 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177), 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:55.910822 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.907390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56225 (local address 127.0.98.2:36177)
0928 17:54:55.907477 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.907479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.907488 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:55.907522 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.907523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.907550 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.907622 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:55.907832 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.908674 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.910032 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.910510 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.910567 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.910620 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.910744 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.910769 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 17:54:55.910851 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.907272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36177 (local address 127.0.98.3:56225)
0928 17:54:55.907332 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.907343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.907386 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.907647 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.907649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.907679 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:55.907767 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.907770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.908784 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.908785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.909934 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.909937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.910588 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.910589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.910668 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.910672 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.910737 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.910776 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:54:55.911185 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8705cd2cc44809b8d44be22e330baa" candidate_uuid: "6383d43b195d437a84b5cbe67f2f9120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732d939282104f8a96b32b50bcfa6309" is_pre_election: true
I20260928 17:54:55.911339 16573 raft_consensus.cc:2454] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6383d43b195d437a84b5cbe67f2f9120 in term 0.
I20260928 17:54:55.911582 16643 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309; no voters:
I20260928 17:54:55.911736 16766 raft_consensus.cc:2781] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:54:55.911823 16766 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:54:55.911859 16766 raft_consensus.cc:3037] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:55.912052 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.907367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38279 (local address 127.0.98.1:44061)
0928 17:54:55.907428 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.907430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.908048 (+ 618us) server_negotiation.cc:407] Connection header received
0928 17:54:55.908120 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.908121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.908134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.908193 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:54:55.909536 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.910107 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.911220 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.911692 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.911778 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.911788 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.911936 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.912005 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:54:55.912287 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.907686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.98.3:38279)
0928 17:54:55.907955 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.907965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.908028 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.908225 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:54:55.908226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:54:55.908228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:54:55.908329 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.908331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.910186 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.910187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.911136 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:54:55.911138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.911779 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.911780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:54:55.911850 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.911854 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:54:55.912172 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 17:54:55.912218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":140,"threads_started":1}
I20260928 17:54:55.912693 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8705cd2cc44809b8d44be22e330baa" candidate_uuid: "6383d43b195d437a84b5cbe67f2f9120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e36336cb5e94aab9fd48dd53a31c500" is_pre_election: true
I20260928 17:54:55.912870 16434 raft_consensus.cc:2454] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6383d43b195d437a84b5cbe67f2f9120 in term 0.
I20260928 17:54:55.912875 16766 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.913017 16766 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [CANDIDATE]: Term 1 election: Requested vote from peers 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177), 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061)
I20260928 17:54:55.913141 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8705cd2cc44809b8d44be22e330baa" candidate_uuid: "6383d43b195d437a84b5cbe67f2f9120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732d939282104f8a96b32b50bcfa6309"
I20260928 17:54:55.913236 16573 raft_consensus.cc:3037] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:55.913234 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8705cd2cc44809b8d44be22e330baa" candidate_uuid: "6383d43b195d437a84b5cbe67f2f9120" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e36336cb5e94aab9fd48dd53a31c500"
I20260928 17:54:55.913295 16434 raft_consensus.cc:3037] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:54:55.914184 16573 raft_consensus.cc:2454] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6383d43b195d437a84b5cbe67f2f9120 in term 1.
I20260928 17:54:55.914355 16643 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309; no voters:
I20260928 17:54:55.914515 16766 raft_consensus.cc:2781] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:54:55.914600 16434 raft_consensus.cc:2454] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6383d43b195d437a84b5cbe67f2f9120 in term 1.
I20260928 17:54:55.914671 16766 raft_consensus.cc:686] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 1 LEADER]: Becoming Leader. State: Replica: 6383d43b195d437a84b5cbe67f2f9120, State: Running, Role: LEADER
I20260928 17:54:55.914769 16766 consensus_queue.cc:237] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:54:55.915611 16291 catalog_manager.cc:5495] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6383d43b195d437a84b5cbe67f2f9120" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } health_report { overall_health: UNKNOWN } } }
I20260928 17:54:55.923417 16761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:54:55.921127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.0.98.3:34939)
0928 17:54:55.921195 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:54:55.921197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:54:55.921206 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:54:55.921283 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:54:55.921284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:54:55.921302 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:54:55.921370 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:54:55.921488 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.921930 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.922664 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:54:55.922872 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:54:55.922946 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:54:55.923095 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:54:55.923200 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:54:55.923292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 17:54:55.923311 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 17:54:55.923329 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:54:55.925161 16434 raft_consensus.cc:1261] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Refusing update from remote peer 6383d43b195d437a84b5cbe67f2f9120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:55.925161 16573 raft_consensus.cc:1261] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Refusing update from remote peer 6383d43b195d437a84b5cbe67f2f9120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:54:55.925511 16768 consensus_queue.cc:1035] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:55.925649 16766 consensus_queue.cc:1035] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:54:55.928726 16773 mvcc.cc:205] Tried to move back new op lower bound from 7334371720907141120 to 7334371720867512320. Current Snapshot: MvccSnapshot[applied={T|T < 7334371720907141120 or (T in {7334371720907141120})}]
I20260928 17:54:55.929653 16291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:55.929749 16291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:54:55.930486 16291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Sending DeleteTablet for 3 replicas of tablet 7a8705cd2cc44809b8d44be22e330baa
I20260928 17:54:55.930792 16689 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8705cd2cc44809b8d44be22e330baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59364
I20260928 17:54:55.930926 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8705cd2cc44809b8d44be22e330baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35142
I20260928 17:54:55.931012 16781 tablet_replica.cc:330] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: stopping tablet replica
I20260928 17:54:55.931036 16553 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8705cd2cc44809b8d44be22e330baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:54:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58146
I20260928 17:54:55.931113 16489 tablet_replica.cc:330] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: stopping tablet replica
I20260928 17:54:55.931149 16781 raft_consensus.cc:2229] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:54:55.931165 16489 raft_consensus.cc:2229] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:55.931226 16489 raft_consensus.cc:2258] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:55.931277 16781 raft_consensus.cc:2258] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:55.931299 16626 tablet_replica.cc:330] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: stopping tablet replica
I20260928 17:54:55.931347 16626 raft_consensus.cc:2229] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:54:55.931401 16626 raft_consensus.cc:2258] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:54:55.931564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16330
I20260928 17:54:55.931700 16781 ts_tablet_manager.cc:1905] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:55.933693 16781 ts_tablet_manager.cc:1918] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:55.933755 16781 log.cc:1196] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/7a8705cd2cc44809b8d44be22e330baa
I20260928 17:54:55.934021 16781 ts_tablet_manager.cc:1939] T 7a8705cd2cc44809b8d44be22e330baa P 6383d43b195d437a84b5cbe67f2f9120: Deleting consensus metadata
I20260928 17:54:55.934577 16276 catalog_manager.cc:4841] TS 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): tablet 7a8705cd2cc44809b8d44be22e330baa (table test-table [id=453929e2cf934e8f9b663074e1af8a6f]) successfully deleted
W20260928 17:54:55.939555 16278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44061 (error 108)
I20260928 17:54:55.939987 16626 ts_tablet_manager.cc:1905] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:54:55.941664 16626 ts_tablet_manager.cc:1918] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:54:55.941728 16626 log.cc:1196] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/7a8705cd2cc44809b8d44be22e330baa
I20260928 17:54:55.941926 16626 ts_tablet_manager.cc:1939] T 7a8705cd2cc44809b8d44be22e330baa P 732d939282104f8a96b32b50bcfa6309: Deleting consensus metadata
I20260928 17:54:55.942525 16277 catalog_manager.cc:4841] TS 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): tablet 7a8705cd2cc44809b8d44be22e330baa (table test-table [id=453929e2cf934e8f9b663074e1af8a6f]) successfully deleted
W20260928 17:54:55.942854 16487 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:54:55.942161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44061 (local address 127.0.0.1:35152)
0928 17:54:55.942222 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:54:55.942233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:54:55.942294 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:54:55.942806 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 17:54:55.943013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16485
W20260928 17:54:55.943080 16278 catalog_manager.cc:4584] TS 3e36336cb5e94aab9fd48dd53a31c500 (127.0.98.1:44061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a8705cd2cc44809b8d44be22e330baa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44061: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:54:55.948128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16624
I20260928 17:54:55.952520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16259
I20260928 17:55:15.959491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--webserver_interface=127.0.98.62
--webserver_port=34461
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:16.046154 16783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:16.046389 16783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:16.046414 16783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:16.047673 16783 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 17:55:16.047722 16783 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 17:55:16.047744 16783 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 17:55:16.047757 16783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:16.047771 16783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:16.047784 16783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:16.047796 16783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:16.047809 16783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:16.049485 16783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:16.049777 16783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:16.050042 16783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:16.055449 16789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.055508 16783 server_base.cc:1034] running on GCE node
W20260928 17:55:16.055466 16791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:16.055464 16788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.055913 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:16.056289 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:16.057439 16783 hybrid_clock.cc:648] HybridClock initialized: now 1790618116057423 us; error 31 us; skew 500 ppm
Sep 28 17:55:16 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:16.064723 16783 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:16.065160 16783 webserver.cc:460] Webserver started at http://127.0.98.62:34461/ using document root <none> and password file <none>
I20260928 17:55:16.065382 16783 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:16.065425 16783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:16.066768 16783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:16.067364 16798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.067530 16783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:16.067631 16783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "0d1df16c213041ec9e75fe01e23a9be7"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "64713f98dff42f751cadfae68dc7165a"
server_key_iv: "df80b6db41f984dc97e0f2bd374ed671"
server_key_version: "encryptionkey@0"
I20260928 17:55:16.068006 16783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:16.076435 16783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:16.078189 16783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:16.078341 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:16.082181 16783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39679
I20260928 17:55:16.082211 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39679 every 8 connection(s)
I20260928 17:55:16.082628 16783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:16.084177 16851 sys_catalog.cc:263] Verifying existing consensus state
I20260928 17:55:16.084591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16783
I20260928 17:55:16.084859 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap starting.
I20260928 17:55:16.084879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44061
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=34753
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:55:16.094128 16851 log.cc:824] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:16.096510 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:55:16.096834 16851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Bootstrap complete.
I20260928 17:55:16.099015 16851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:55:16.099256 16851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Initialized, Role: FOLLOWER
I20260928 17:55:16.099435 16851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:55:16.099510 16851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:16.099560 16851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:16.099611 16851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 2 FOLLOWER]: Advancing to term 3
I20260928 17:55:16.100656 16851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:55:16.100780 16851 leader_election.cc:302] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1df16c213041ec9e75fe01e23a9be7; no voters:
I20260928 17:55:16.100975 16851 leader_election.cc:288] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 17:55:16.101011 16855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 17:55:16.101126 16855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [term 3 LEADER]: Becoming Leader. State: Replica: 0d1df16c213041ec9e75fe01e23a9be7, State: Running, Role: LEADER
I20260928 17:55:16.101231 16851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:16.101318 16855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } }
I20260928 17:55:16.101611 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1df16c213041ec9e75fe01e23a9be7. Latest consensus state: current_term: 3 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:55:16.101616 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0d1df16c213041ec9e75fe01e23a9be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1df16c213041ec9e75fe01e23a9be7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39679 } } }
I20260928 17:55:16.101773 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:16.102018 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:16.102188 16864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:16.103108 16864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:16.103365 16864 catalog_manager.cc:1259] Loaded cluster ID: 16d0b983882f4574a2915067c635c01b
I20260928 17:55:16.103493 16864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:16.104786 16864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:16.104964 16864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Loaded TSK: 0
I20260928 17:55:16.105013 16864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d1df16c213041ec9e75fe01e23a9be7: Loaded TSK: 1
F20260928 17:55:16.105261 16864 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff8c27749ad google::LogMessage::Fail() at ??:0
@ 0x7ff8c277788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff8c27744de google::LogMessage::Flush() at ??:0
@ 0x7ff8c2775a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff8c4ee1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff8c4eded41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff8ca6d21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff8ca6d248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff8ca6d2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff8ca698c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff8ca6a2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff8c46420d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff8c4637954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff8c549b6db start_thread at ??:0
@ 0x7ff8c127261f clone at ??:0
W20260928 17:55:16.187261 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:16.187510 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:16.187552 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:16.188844 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:16.188905 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:16.188951 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:16.190479 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34753
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:16.190747 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:16.191138 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:16.196177 16893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:16.196177 16895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.196228 16853 server_base.cc:1034] running on GCE node
W20260928 17:55:16.196322 16892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.196590 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:16.196861 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:16.198014 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790618116197995 us; error 43 us; skew 500 ppm
Sep 28 17:55:16 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:16.200577 16853 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:55:16.201097 16853 webserver.cc:460] Webserver started at http://127.0.98.1:34753/ using document root <none> and password file <none>
I20260928 17:55:16.201337 16853 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:16.201402 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:16.202757 16853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:16.203426 16902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.203575 16853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:16.203661 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "3e36336cb5e94aab9fd48dd53a31c500"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "776d1ad24f3b1e2b858fbb5dc7f154fd"
server_key_iv: "437a690fd8271a1d3c9cfef3b3e1a14e"
server_key_version: "encryptionkey@0"
I20260928 17:55:16.204018 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:16.218233 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:16.220808 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:16.220983 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:16.221230 16853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:16.221668 16909 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 17:55:16.223796 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 17:55:16.223861 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 17:55:16.223905 16853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 17:55:16.224606 16853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 17:55:16.224658 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 17:55:16.224733 16909 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap starting.
I20260928 17:55:16.232012 16853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44061
I20260928 17:55:16.232197 17016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44061 every 8 connection(s)
I20260928 17:55:16.232475 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:55:16.236037 16909 log.cc:824] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:16.238422 16909 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 17:55:16.238714 16909 tablet_bootstrap.cc:492] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Bootstrap complete.
I20260928 17:55:16.239639 16909 ts_tablet_manager.cc:1397] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260928 17:55:16.240573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 16853
I20260928 17:55:16.240873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36177
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=43295
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:55:16.241101 16909 raft_consensus.cc:348] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:16.241487 16909 raft_consensus.cc:729] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e36336cb5e94aab9fd48dd53a31c500, State: Initialized, Role: FOLLOWER
I20260928 17:55:16.241657 16909 consensus_queue.cc:260] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:16.241957 16909 ts_tablet_manager.cc:1428] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 17:55:16.313359 17019 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:55:16.233147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:51081)
0928 17:55:16.233325 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.233370 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:16.233834 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:16.313149 (+ 79315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
W20260928 17:55:16.313665 17017 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39679: Client connection negotiation failed: client connection to 127.0.98.62:39679: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 17:55:16.314046 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.313912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:60937)
0928 17:55:16.313967 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.314017 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:55:16.314373 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.314275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:44737)
0928 17:55:16.314323 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.314352 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:55:16.314510 17017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 17:55:16.323597 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:16.323817 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:16.323839 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:16.325161 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:16.325201 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:16.325227 17022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:16.326719 17022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43295
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:16.326948 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:16.327183 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:16.332473 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:16.332473 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:16.332571 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.332787 17022 server_base.cc:1034] running on GCE node
I20260928 17:55:16.333033 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:16.333288 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:16.334452 17022 hybrid_clock.cc:648] HybridClock initialized: now 1790618116334438 us; error 28 us; skew 500 ppm
Sep 28 17:55:16 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:16.337091 17022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:55:16.337553 17022 webserver.cc:460] Webserver started at http://127.0.98.2:43295/ using document root <none> and password file <none>
I20260928 17:55:16.337790 17022 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:16.337837 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:16.339212 17022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:16.339775 17037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.339948 17022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:16.340046 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "732d939282104f8a96b32b50bcfa6309"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "33ba2c7877fe93b43950a0d1b4bfb4a1"
server_key_iv: "31d1a30f480901626d6b3204e526eb9b"
server_key_version: "encryptionkey@0"
I20260928 17:55:16.340449 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:16.350198 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:16.352439 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:16.352588 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:16.352833 17022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:16.353116 17022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:16.353163 17022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.353201 17022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:16.353232 17022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.359357 17022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36177
I20260928 17:55:16.359404 17150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36177 every 8 connection(s)
I20260928 17:55:16.359759 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:55:16.360822 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.360451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:60063)
0928 17:55:16.360604 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.360672 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20260928 17:55:16.361199 17151 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39679: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260928 17:55:16.361445 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.361379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:49239)
0928 17:55:16.361405 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.361429 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 17:55:16.361657 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.361609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:41033)
0928 17:55:16.361629 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.361646 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 17:55:16.361754 17151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:55:16.366195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17022
I20260928 17:55:16.366493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34939
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39679
--webserver_port=44565
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:16.450350 17154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:16.450565 17154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:16.450589 17154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:16.451786 17154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:16.451831 17154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:16.451860 17154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:16.453342 17154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44565
--tserver_master_addrs=127.0.98.62:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:16.453547 17154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:16.453739 17154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:16.458838 17159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.458956 17154 server_base.cc:1034] running on GCE node
W20260928 17:55:16.458858 17160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:16.459064 17162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:16.459292 17154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:16.459543 17154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:16.460700 17154 hybrid_clock.cc:648] HybridClock initialized: now 1790618116460692 us; error 34 us; skew 500 ppm
Sep 28 17:55:16 dist-test-slave-zqlt krb5kdc[15714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:16.463248 17154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:55:16.463671 17154 webserver.cc:460] Webserver started at http://127.0.98.3:44565/ using document root <none> and password file <none>
I20260928 17:55:16.463912 17154 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:16.463979 17154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:16.465402 17154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:16.465968 17169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.466128 17154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:16.466221 17154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "6383d43b195d437a84b5cbe67f2f9120"
format_stamp: "Formatted at 2026-09-28 17:54:34 on dist-test-slave-zqlt"
server_key: "7889f5f0315dedbce388b0d5d33da7f8"
server_key_iv: "ba23c41cd3e603b886e9ebe2aed06a1c"
server_key_version: "encryptionkey@0"
I20260928 17:55:16.466590 17154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:16.479161 17154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:16.481617 17154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:16.481784 17154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:16.482018 17154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:16.482333 17154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:16.482393 17154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.482429 17154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:16.482455 17154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:16.488538 17154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34939
I20260928 17:55:16.488632 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34939 every 8 connection(s)
I20260928 17:55:16.488935 17154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:55:16.489909 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.489561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:34893)
0928 17:55:16.489707 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.489776 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
W20260928 17:55:16.490284 17283 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39679 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39679: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
I20260928 17:55:16.490546 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.490466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:40405)
0928 17:55:16.490505 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.490530 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 17:55:16.490773 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:16.490725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:52751)
0928 17:55:16.490746 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:16.490763 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 17:55:16.490883 17283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 17:55:16.491339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17154
I20260928 17:55:17.315192 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.314934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:42725)
0928 17:55:17.315088 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:17.315126 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260928 17:55:17.362291 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.362085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:57231)
0928 17:55:17.362201 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:17.362239 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260928 17:55:17.491456 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.491224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:55917)
0928 17:55:17.491362 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:17.491404 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 17:55:17.704694 17290 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:17.704816 17290 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:17.705228 17290 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
Sep 28 17:55:17 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 17:55:17 dist-test-slave-zqlt krb5kdc[15714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 17:55:17.715988 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.705528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40361 (local address 127.0.98.3:34939)
0928 17:55:17.705686 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:17.705699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:17.705736 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:55:17.706217 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:17.706220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:17.706263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:17.706420 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:17.707412 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.708455 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709463 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.709864 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:17.712402 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:17.712406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:17.712406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:17.712418 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:17.714231 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:17.714723 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:17.714725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:17.714726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:17.714815 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:17.715079 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:17.715080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:17.715081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:17.715337 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:17.715424 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:17.715731 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 17:55:17.715860 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260928 17:55:17.716003 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.705405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34939 (local address 127.0.98.1:40361)
0928 17:55:17.705481 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:17.705494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:17.705555 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:17.706453 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:17.706456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:17.706991 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:17.707281 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:17.707289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.708543 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.708544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:17.709360 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:17.709362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.709957 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:17.710066 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:17.710720 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:17.710730 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:17.712297 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:17.714316 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:17.714323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:17.714325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:17.714628 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:17.714901 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:17.714902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:17.714902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:17.715014 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:17.715417 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:17.715424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:17.715561 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 17:55:17.715671 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 17:55:17.715862 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":23}
I20260928 17:55:17.716122 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.705548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53683 (local address 127.0.98.2:36177)
0928 17:55:17.705756 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:17.705765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:17.705790 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:55:17.706218 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:17.706220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:17.706263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:17.706420 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:17.707430 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.708417 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709463 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.709865 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709983 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:17.712776 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:17.712780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:17.712781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:17.712792 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:17.714768 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:17.715079 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:17.715082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:17.715083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:17.715171 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:17.715379 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:17.715380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:17.715380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:17.715598 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:17.715695 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:17.715850 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:55:17.715980 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260928 17:55:17.716451 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:17.705438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36177 (local address 127.0.98.1:53683)
0928 17:55:17.705561 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:17.705572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:17.705627 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:17.706453 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:17.706456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:17.706990 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:17.707281 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:17.707289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.708519 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.708520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:17.709359 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:17.709362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:17.709953 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:17.709954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:17.710067 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:17.710684 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:17.710699 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:17.712672 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:17.714841 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:17.714842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:17.714843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:17.715002 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:17.715248 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:17.715249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:17.715249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:17.715312 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:17.715676 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:17.715677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:17.715723 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:55:17.715791 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 17:55:17.715928 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":85,"thread_start_us":58,"threads_started":1}
W20260928 17:55:17.717325 16904 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
W20260928 17:55:17.717432 16903 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
I20260928 17:55:17.717458 16903 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500; no voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309
I20260928 17:55:17.717612 17290 raft_consensus.cc:2726] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:55:18.315939 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:18.315687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:42455)
0928 17:55:18.315840 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:18.315888 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260928 17:55:18.363026 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:18.362803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:52851)
0928 17:55:18.362941 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:18.362984 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260928 17:55:18.492043 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:18.491863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:34919)
0928 17:55:18.491973 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:18.492003 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260928 17:55:19.316735 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:19.316521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:33471)
0928 17:55:19.316653 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:19.316691 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260928 17:55:19.363786 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:19.363556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:42833)
0928 17:55:19.363689 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:19.363727 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 17:55:19.492705 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:19.492498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:36853)
0928 17:55:19.492626 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:19.492660 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260928 17:55:19.628373 17300 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:19.628494 17300 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:19.628661 17300 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
W20260928 17:55:19.629068 16903 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
W20260928 17:55:19.629143 16904 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
I20260928 17:55:19.629184 16904 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500; no voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309
I20260928 17:55:19.629268 17300 raft_consensus.cc:2726] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:55:20.317476 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:20.317235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:58337)
0928 17:55:20.317383 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:20.317427 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260928 17:55:20.364564 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:20.364303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:45703)
0928 17:55:20.364473 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:20.364520 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260928 17:55:20.493426 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:20.493223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:45517)
0928 17:55:20.493343 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:20.493381 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 17:55:21.318212 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:21.317980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:33609)
0928 17:55:21.318125 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:21.318163 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260928 17:55:21.365320 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:21.365100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:46763)
0928 17:55:21.365229 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:21.365269 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260928 17:55:21.494086 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:21.493883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:34049)
0928 17:55:21.494009 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:21.494045 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260928 17:55:22.318964 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:22.318756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:50361)
0928 17:55:22.318878 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:22.318917 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 17:55:22.366067 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:22.365837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:47787)
0928 17:55:22.365983 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:22.366024 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260928 17:55:22.494683 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:22.494502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:38265)
0928 17:55:22.494613 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:22.494646 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 17:55:22.962815 17310 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:22.962919 17310 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:22.963143 17310 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
W20260928 17:55:22.963572 16903 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
W20260928 17:55:22.963672 16904 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
I20260928 17:55:22.963771 16904 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500; no voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309
I20260928 17:55:22.963860 17310 raft_consensus.cc:2726] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:55:23.319697 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:23.319482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:51767)
0928 17:55:23.319607 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:23.319649 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 17:55:23.366922 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:23.366642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:46095)
0928 17:55:23.366833 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:23.366877 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":63,"threads_started":1}
I20260928 17:55:23.495273 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:23.495078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:60935)
0928 17:55:23.495202 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:23.495233 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260928 17:55:24.320473 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:24.320239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:32983)
0928 17:55:24.320387 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:24.320424 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260928 17:55:24.367712 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:24.367463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:36909)
0928 17:55:24.367621 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:24.367668 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260928 17:55:24.495920 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:24.495699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:55649)
0928 17:55:24.495840 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:24.495877 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 17:55:25.321233 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:25.320971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:59315)
0928 17:55:25.321147 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:25.321182 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260928 17:55:25.368499 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:25.368283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:57811)
0928 17:55:25.368411 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:25.368454 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 17:55:25.496505 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:25.496304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:44287)
0928 17:55:25.496427 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:25.496460 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260928 17:55:25.871857 17320 raft_consensus.cc:482] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:25.871958 17320 raft_consensus.cc:504] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6383d43b195d437a84b5cbe67f2f9120" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34939 } } peers { permanent_uuid: "732d939282104f8a96b32b50bcfa6309" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36177 } } peers { permanent_uuid: "3e36336cb5e94aab9fd48dd53a31c500" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44061 } }
I20260928 17:55:25.872139 17320 leader_election.cc:288] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939), 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177)
W20260928 17:55:25.872529 16903 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6383d43b195d437a84b5cbe67f2f9120 (127.0.98.3:34939): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
W20260928 17:55:25.872661 16904 leader_election.cc:341] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 732d939282104f8a96b32b50bcfa6309 (127.0.98.2:36177): Not found: Tablet not found: 7a8705cd2cc44809b8d44be22e330baa
I20260928 17:55:25.872725 16904 leader_election.cc:302] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e36336cb5e94aab9fd48dd53a31c500; no voters: 6383d43b195d437a84b5cbe67f2f9120, 732d939282104f8a96b32b50bcfa6309
I20260928 17:55:25.872812 17320 raft_consensus.cc:2726] T 7a8705cd2cc44809b8d44be22e330baa P 3e36336cb5e94aab9fd48dd53a31c500 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 17:55:26.322098 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:26.321785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:36697)
0928 17:55:26.322005 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:26.322048 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260928 17:55:26.369220 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:26.368980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:54139)
0928 17:55:26.369129 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:26.369172 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 17:55:26.497294 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:26.497044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:37021)
0928 17:55:26.497203 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:26.497249 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260928 17:55:27.322939 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:27.322681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:36475)
0928 17:55:27.322841 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:27.322888 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260928 17:55:27.369940 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:27.369726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:41613)
0928 17:55:27.369851 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:27.369896 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260928 17:55:27.497901 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:27.497736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:33791)
0928 17:55:27.497838 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:27.497866 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260928 17:55:28.323735 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:28.323476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:35691)
0928 17:55:28.323640 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:28.323685 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260928 17:55:28.370734 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:28.370534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:59043)
0928 17:55:28.370654 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:28.370689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260928 17:55:28.498507 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:28.498284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:57557)
0928 17:55:28.498422 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:28.498461 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260928 17:55:29.324494 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:29.324252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:36155)
0928 17:55:29.324410 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:29.324446 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 17:55:29.371461 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:29.371199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:37255)
0928 17:55:29.371360 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:29.371414 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260928 17:55:29.499205 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:29.499000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:37769)
0928 17:55:29.499124 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:29.499161 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260928 17:55:30.325248 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:30.324992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:48025)
0928 17:55:30.325159 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:30.325200 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260928 17:55:30.372308 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:30.372069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:52883)
0928 17:55:30.372220 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:30.372264 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 17:55:30.500064 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:30.499819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:51189)
0928 17:55:30.499969 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:30.500017 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260928 17:55:31.326128 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:31.325909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.1:43619)
0928 17:55:31.326046 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:31.326080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 17:55:31.373178 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:31.372944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.2:53667)
0928 17:55:31.373089 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:31.373132 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260928 17:55:31.495458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 16853
I20260928 17:55:31.500229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17022
I20260928 17:55:31.501411 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:31.501200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39679 (local address 127.0.98.3:47083)
0928 17:55:31.501321 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:31.501352 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":50,"threads_started":1}
I20260928 17:55:31.504671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17154
2026-09-28T17:55:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58554 ms)
[----------] 30 tests from SecurityITest (233265 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:31 dist-test-slave-zqlt krb5kdc[17341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:31 dist-test-slave-zqlt krb5kdc[17341](info): set up 2 sockets
Sep 28 17:55:31 dist-test-slave-zqlt krb5kdc[17341](info): commencing operation
krb5kdc: starting...
W20260928 17:55:32.531901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:32 dist-test-slave-zqlt krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:32.654572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:32.736763 17354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:32.737025 17354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:32.737061 17354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:32.738308 17354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:32.738381 17354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:32.738405 17354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:32.738435 17354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:32.738467 17354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:32.739961 17354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:32.740270 17354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:32.740509 17354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:32.745254 17360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:32.745318 17359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:32.745478 17354 server_base.cc:1034] running on GCE node
W20260928 17:55:32.745254 17362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:32.745713 17354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:32.746052 17354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:32.747231 17354 hybrid_clock.cc:648] HybridClock initialized: now 1790618132747195 us; error 42 us; skew 500 ppm
Sep 28 17:55:32 dist-test-slave-zqlt krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:32.749775 17354 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:32.750295 17354 webserver.cc:460] Webserver started at http://127.0.98.62:46769/ using document root <none> and password file <none>
I20260928 17:55:32.750528 17354 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:32.750597 17354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:32.750761 17354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:32.751866 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "84e782f07d1245b1a8d53966b7557600"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "6a61113e125f128dc7eb6a530c508fc0"
server_key_iv: "a82d7cf2810fad006514aef807e18c55"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.752312 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "84e782f07d1245b1a8d53966b7557600"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "6a61113e125f128dc7eb6a530c508fc0"
server_key_iv: "a82d7cf2810fad006514aef807e18c55"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.753682 17354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:32.754407 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.754612 17354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.754711 17354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "84e782f07d1245b1a8d53966b7557600"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "6a61113e125f128dc7eb6a530c508fc0"
server_key_iv: "a82d7cf2810fad006514aef807e18c55"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.754796 17354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:32.765491 17354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:32.767766 17354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:32.767926 17354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:32.771940 17354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40797
I20260928 17:55:32.771976 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40797 every 8 connection(s)
I20260928 17:55:32.772356 17354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:32.773257 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:32.775521 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: Bootstrap starting.
I20260928 17:55:32.776160 17422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:32.776453 17422 log.cc:824] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:32.777129 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: No bootstrap required, opened a new log
I20260928 17:55:32.778692 17422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260928 17:55:32.778841 17422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:32.778864 17422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e782f07d1245b1a8d53966b7557600, State: Initialized, Role: FOLLOWER
I20260928 17:55:32.779032 17422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260928 17:55:32.779096 17422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:32.779117 17422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:32.779143 17422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:32.779498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17354
I20260928 17:55:32.779624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:32.779858 392 external_mini_cluster.cc:1428] Setting key 404b3b14387538a7edc14079267aa5ea
I20260928 17:55:32.779953 17422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260928 17:55:32.780059 17422 leader_election.cc:302] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84e782f07d1245b1a8d53966b7557600; no voters:
I20260928 17:55:32.780256 17422 leader_election.cc:288] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:32.780274 17425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:32.780447 17425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [term 1 LEADER]: Becoming Leader. State: Replica: 84e782f07d1245b1a8d53966b7557600, State: Running, Role: LEADER
I20260928 17:55:32.780571 17422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:32.780552 17425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260928 17:55:32.780885 17425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84e782f07d1245b1a8d53966b7557600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260928 17:55:32.780959 17425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:32.781229 17432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:32.781286 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84e782f07d1245b1a8d53966b7557600. Latest consensus state: current_term: 1 leader_uuid: "84e782f07d1245b1a8d53966b7557600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e782f07d1245b1a8d53966b7557600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260928 17:55:32.781456 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:32.781898 17432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:32.783550 17432 catalog_manager.cc:1347] Generated new cluster ID: 97d4cf1616044fca9ac80897de4e3433
I20260928 17:55:32.783599 17432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:32 dist-test-slave-zqlt krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:32.794677 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:32.780417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.98.62:40797)
0928 17:55:32.780619 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:32.780629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:32.782261 (+ 1632us) server_negotiation.cc:407] Connection header received
0928 17:55:32.782643 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:32.782652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:32.782711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:32.782859 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:32.783774 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.785197 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.786363 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.787011 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.787095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:32.790180 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:32.790184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:32.790185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:32.790197 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:32.792973 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:32.793385 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:32.793389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:32.793389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:32.793526 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:32.793733 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:32.793734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:32.793734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:32.794045 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:32.794217 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:32.794309 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 17:55:32.794451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260928 17:55:32.796411 17432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:32.796938 17432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:32.801585 17432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: Generated new TSK 0
I20260928 17:55:32.801764 17432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:55:32.830821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:32.914527 17448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:32.914752 17448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:32.914778 17448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:32.916142 17448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:32.916193 17448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:32.916222 17448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:32.917825 17448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:32.918099 17448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:32.918352 17448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:32.923555 17454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:32.923568 17453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:32.923563 17456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:32.923679 17448 server_base.cc:1034] running on GCE node
I20260928 17:55:32.924023 17448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:32.924327 17448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:32.925513 17448 hybrid_clock.cc:648] HybridClock initialized: now 1790618132925496 us; error 27 us; skew 500 ppm
Sep 28 17:55:32 dist-test-slave-zqlt krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:32.928325 17448 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:55:32.928726 17448 webserver.cc:460] Webserver started at http://127.0.98.1:38353/ using document root <none> and password file <none>
I20260928 17:55:32.928920 17448 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:32.928960 17448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:32.929049 17448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:32.930084 17448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "9c811981adc64e1aa89db56150947fa6"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "f77eb0e2e38e55e68a6fe8d6f56e01a5"
server_key_iv: "976205cdaddd2ae469e726f5c90a97fa"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.930399 17448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9c811981adc64e1aa89db56150947fa6"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "f77eb0e2e38e55e68a6fe8d6f56e01a5"
server_key_iv: "976205cdaddd2ae469e726f5c90a97fa"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.931777 17448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 17:55:32.932526 17463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.932677 17448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.932765 17448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "9c811981adc64e1aa89db56150947fa6"
format_stamp: "Formatted at 2026-09-28 17:55:32 on dist-test-slave-zqlt"
server_key: "f77eb0e2e38e55e68a6fe8d6f56e01a5"
server_key_iv: "976205cdaddd2ae469e726f5c90a97fa"
server_key_version: "encryptionkey@0"
I20260928 17:55:32.932837 17448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:32.958482 17448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:32.960855 17448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:32.961004 17448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:32.961263 17448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:32.961583 17448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:32.961616 17448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.961659 17448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:32.961688 17448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:32.967691 17448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32933
I20260928 17:55:32.967742 17576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32933 every 8 connection(s)
I20260928 17:55:32.968068 17448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:55:32 dist-test-slave-zqlt krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:32.976012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17448
I20260928 17:55:32.976161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:32.976374 392 external_mini_cluster.cc:1428] Setting key dd549ac8c9a47fcca045c2fcdf442b8f
I20260928 17:55:32.977409 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:32.968803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55955 (local address 127.0.98.62:40797)
0928 17:55:32.968892 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:32.968894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:32.969067 (+ 173us) server_negotiation.cc:407] Connection header received
0928 17:55:32.970015 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:32.970016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:32.970029 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:32.970078 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:32.970691 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.971188 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.972308 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.972514 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.972572 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:32.974801 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:32.974802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:32.974803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:32.974810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:32.976021 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:32.976445 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:32.976447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:32.976447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:32.976510 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:32.976765 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:32.976766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:32.976766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:32.976881 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:32.976961 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:32.977227 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 17:55:32.977323 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 17:55:32.977599 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:32.968752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.1:55955)
0928 17:55:32.968933 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:32.968970 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:32.969914 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:32.970140 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:32.970144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:32.970411 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:32.970611 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:32.970615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.971272 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.971273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:32.972197 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:32.972200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:32.972591 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:32.972592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:32.972698 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:32.973306 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:32.973315 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:32.974702 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:32.976123 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:32.976131 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:32.976134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:32.976364 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:32.976580 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:32.976581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:32.976581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:32.976686 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:32.976950 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:32.976957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:32.977095 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 17:55:32.977205 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 17:55:32.977356 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260928 17:55:32.978071 17577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260928 17:55:32.978183 17577 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:32.978382 17577 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260928 17:55:32.978912 17386 ts_manager.cc:194] Registered new tserver with Master: 9c811981adc64e1aa89db56150947fa6 (127.0.98.1:32933)
I20260928 17:55:32.979518 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55955
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:55:33.003585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:33.088809 17582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.089054 17582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.089093 17582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.090261 17582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.090328 17582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.090371 17582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:33.091884 17582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.092164 17582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.092403 17582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:33.097543 17587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.097569 17588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.097617 17582 server_base.cc:1034] running on GCE node
W20260928 17:55:33.097573 17590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.098029 17582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.098320 17582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.099471 17582 hybrid_clock.cc:648] HybridClock initialized: now 1790618133099450 us; error 36 us; skew 500 ppm
Sep 28 17:55:33 dist-test-slave-zqlt krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:33.102154 17582 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:55:33.102599 17582 webserver.cc:460] Webserver started at http://127.0.98.2:46257/ using document root <none> and password file <none>
I20260928 17:55:33.102842 17582 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.102912 17582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.103066 17582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.104216 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "f28bd1d12c6740a88c380baa3172ffa2"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "77872bd3a0b57b77e365fd2ac456efc6"
server_key_iv: "b2b24e2d58912a4f72ffc62066e1f332"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.104585 17582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f28bd1d12c6740a88c380baa3172ffa2"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "77872bd3a0b57b77e365fd2ac456efc6"
server_key_iv: "b2b24e2d58912a4f72ffc62066e1f332"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.105988 17582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:55:33.106786 17597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.107129 17582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:33.107250 17582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "f28bd1d12c6740a88c380baa3172ffa2"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "77872bd3a0b57b77e365fd2ac456efc6"
server_key_iv: "b2b24e2d58912a4f72ffc62066e1f332"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.107342 17582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.137233 17582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.139832 17582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.140012 17582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.140303 17582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:33.140630 17582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:33.140686 17582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.140750 17582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:33.140780 17582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.146870 17582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42461
I20260928 17:55:33.146914 17710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42461 every 8 connection(s)
I20260928 17:55:33.147262 17582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:55:33.149120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17582
I20260928 17:55:33.149271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:33.149513 392 external_mini_cluster.cc:1428] Setting key 5dad01f98a9f515dc94fd700ee7cc5ec
Sep 28 17:55:33 dist-test-slave-zqlt krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:33.158046 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.148047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39455 (local address 127.0.98.62:40797)
0928 17:55:33.148150 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.148153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.148337 (+ 184us) server_negotiation.cc:407] Connection header received
0928 17:55:33.149381 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.149383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.149399 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.149487 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.150165 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.150918 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.152338 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.152528 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.152591 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.154928 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.154929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.154930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:33.154938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.156490 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.156928 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.156930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.156931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.156988 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.157305 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.157306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.157306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.157471 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.157560 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.157810 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.157911 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 17:55:33.158102 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.147996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.2:39455)
0928 17:55:33.148198 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.148244 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.149198 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.149549 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.149553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.149871 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:33.150078 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.150083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.151025 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.151025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.152211 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.152223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.152618 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.152619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.152776 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.153329 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:33.153337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:33.154829 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:33.156572 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:33.156579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:33.156583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:33.156817 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:33.157080 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:33.157081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:33.157081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:33.157201 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:33.157549 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:33.157553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:33.157673 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.157757 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.157888 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 17:55:33.158674 17711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260928 17:55:33.158762 17711 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:33.158914 17711 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260928 17:55:33.159338 17386 ts_manager.cc:194] Registered new tserver with Master: f28bd1d12c6740a88c380baa3172ffa2 (127.0.98.2:42461)
I20260928 17:55:33.159716 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:55:33.178642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:33.260134 17716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.260385 17716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.260422 17716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.261618 17716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.261696 17716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.261742 17716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:33.263309 17716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.263588 17716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.263826 17716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:33.269110 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.269268 17716 server_base.cc:1034] running on GCE node
W20260928 17:55:33.269299 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.269114 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.269534 17716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.269752 17716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.270901 17716 hybrid_clock.cc:648] HybridClock initialized: now 1790618133270884 us; error 40 us; skew 500 ppm
Sep 28 17:55:33 dist-test-slave-zqlt krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:33.273663 17716 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:55:33.274145 17716 webserver.cc:460] Webserver started at http://127.0.98.3:46837/ using document root <none> and password file <none>
I20260928 17:55:33.274399 17716 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.274468 17716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.274614 17716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.275684 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "2258da473f40406bbc85f123848a6918"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "21dbade85bbcfd083af973cc361af696"
server_key_iv: "2a8d82766a84e1ea9c102dd6c8e858c0"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.276070 17716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2258da473f40406bbc85f123848a6918"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "21dbade85bbcfd083af973cc361af696"
server_key_iv: "2a8d82766a84e1ea9c102dd6c8e858c0"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.277439 17716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:33.278206 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.278405 17716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:33.278513 17716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "2258da473f40406bbc85f123848a6918"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "21dbade85bbcfd083af973cc361af696"
server_key_iv: "2a8d82766a84e1ea9c102dd6c8e858c0"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.278600 17716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.297006 17716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.299557 17716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.299763 17716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.300040 17716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:33.300419 17716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:33.300480 17716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.300529 17716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:33.300573 17716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.307185 17716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43085
I20260928 17:55:33.307232 17844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43085 every 8 connection(s)
I20260928 17:55:33.307600 17716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:55:33.314020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17716
I20260928 17:55:33.314108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:33.314327 392 external_mini_cluster.cc:1428] Setting key 0bf187c27196d72210d359e61c30dcbc
Sep 28 17:55:33 dist-test-slave-zqlt krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:33.317982 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.308420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56241 (local address 127.0.98.62:40797)
0928 17:55:33.308500 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.308503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.308831 (+ 328us) server_negotiation.cc:407] Connection header received
0928 17:55:33.309795 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.309796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.309807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.309866 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.310702 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.311294 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.312389 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.312583 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.312640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.314886 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.314888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.314889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:33.314897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.316566 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.317009 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.317011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.317011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.317073 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.317320 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.317321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.317321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.317442 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.317524 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.317813 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.317890 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 17:55:33.318183 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.308397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40797 (local address 127.0.98.3:56241)
0928 17:55:33.308607 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.308671 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.309683 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.309946 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.309949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.310393 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:33.310602 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.310607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.311387 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.311389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.312309 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.312312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.312657 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.312658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.312754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.313281 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:33.313289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:33.314781 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:33.316671 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:33.316678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:33.316680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:33.316917 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:33.317139 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:33.317140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:33.317140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:33.317251 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:33.317518 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:33.317524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:33.317667 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.317790 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.317952 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260928 17:55:33.318681 17845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40797
I20260928 17:55:33.318790 17845 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:33.318965 17845 heartbeater.cc:511] Master 127.0.98.62:40797 requested a full tablet report, sending...
I20260928 17:55:33.319363 17386 ts_manager.cc:194] Registered new tserver with Master: 2258da473f40406bbc85f123848a6918 (127.0.98.3:43085)
I20260928 17:55:33.319754 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56241
I20260928 17:55:33.326334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:33.333511 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.327803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.98.62:40797)
0928 17:55:33.327863 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.327865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.327934 (+ 69us) server_negotiation.cc:407] Connection header received
0928 17:55:33.327975 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.327976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.327985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.328035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.328643 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.329140 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.329856 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.330033 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.330084 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.331209 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.331210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.331211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:33.331217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.332531 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.332857 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.332858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.332858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.332908 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:33.333070 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:33.333071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:33.333071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:33.333216 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.333292 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.333384 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.333444 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:55:33.335367 17386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:33.337152 17386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:33.344342 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.339789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.98.1:32933)
0928 17:55:33.340310 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.340318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.340339 (+ 21us) server_negotiation.cc:407] Connection header received
0928 17:55:33.340389 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.340392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.340411 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.340501 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:33.340804 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341558 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.343247 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.343872 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.343967 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344038 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.344207 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.344251 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":101,"threads_started":1}
I20260928 17:55:33.344348 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.339682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32933 (local address 127.0.0.1:39988)
0928 17:55:33.340032 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.340044 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.340122 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.340607 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.340608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.340611 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:33.340686 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.340738 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341674 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.341675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.343140 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.343143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.343978 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.343979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.344068 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344083 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.344196 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.344248 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":35,"thread_start_us":96,"threads_started":1}
I20260928 17:55:33.344518 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.340031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.0.98.2:42461)
0928 17:55:33.340347 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.340352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.340361 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:55:33.340403 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.340406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.340421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.340501 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:33.340760 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341852 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.343323 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.343973 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.344037 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.344394 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.344437 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
I20260928 17:55:33.344360 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.339574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43085 (local address 127.0.0.1:45202)
0928 17:55:33.339982 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.340018 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.340142 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.340556 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.340559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.340562 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:33.340641 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.340643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341994 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.341995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.343104 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.343107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.344045 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.344046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.344135 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344140 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.344171 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.344239 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260928 17:55:33.344751 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.339682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.0.98.3:43085)
0928 17:55:33.340398 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.340403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.340413 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:55:33.340454 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.340456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.340473 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.340551 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:33.340700 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341870 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.343190 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.343937 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.344400 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344467 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.344491 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.344643 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":99,"threads_started":1}
I20260928 17:55:33.344820 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.339829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42461 (local address 127.0.0.1:52650)
0928 17:55:33.340233 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.340241 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.340281 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.340609 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.340610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.340611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:33.340686 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.340688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.341967 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.341968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.343244 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.343246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.344144 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.344145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.344198 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.344201 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.344257 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.344292 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":45,"thread_start_us":66,"threads_started":1}
I20260928 17:55:33.345407 17779 tablet_service.cc:1467] Processing CreateTablet for tablet b476e5da78194af6ad91d186937eded7 (DEFAULT_TABLE table=test-table [id=d366c0a800544770b0c12c0cdf3f53da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:33.345407 17511 tablet_service.cc:1467] Processing CreateTablet for tablet b476e5da78194af6ad91d186937eded7 (DEFAULT_TABLE table=test-table [id=d366c0a800544770b0c12c0cdf3f53da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:33.345731 17779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e5da78194af6ad91d186937eded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:33.345803 17511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e5da78194af6ad91d186937eded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:33.348868 17863 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: Bootstrap starting.
I20260928 17:55:33.349449 17645 tablet_service.cc:1467] Processing CreateTablet for tablet b476e5da78194af6ad91d186937eded7 (DEFAULT_TABLE table=test-table [id=d366c0a800544770b0c12c0cdf3f53da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:33.349670 17863 tablet_bootstrap.cc:654] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:33.349836 17645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b476e5da78194af6ad91d186937eded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:33.350360 17863 log.cc:824] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:33.350749 17862 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Bootstrap starting.
I20260928 17:55:33.351488 17862 tablet_bootstrap.cc:654] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:33.352137 17862 log.cc:824] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:33.352564 17863 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: No bootstrap required, opened a new log
I20260928 17:55:33.352661 17863 ts_tablet_manager.cc:1397] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 17:55:33.353006 17862 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: No bootstrap required, opened a new log
I20260928 17:55:33.353085 17862 ts_tablet_manager.cc:1397] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:33.353837 17866 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Bootstrap starting.
I20260928 17:55:33.354669 17866 tablet_bootstrap.cc:654] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:33.354771 17863 raft_consensus.cc:348] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.354991 17863 raft_consensus.cc:374] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:33.355043 17863 raft_consensus.cc:729] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c811981adc64e1aa89db56150947fa6, State: Initialized, Role: FOLLOWER
I20260928 17:55:33.355177 17863 consensus_queue.cc:260] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.355257 17866 log.cc:824] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:33.355504 17863 ts_tablet_manager.cc:1428] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:33.355388 17862 raft_consensus.cc:348] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.355581 17862 raft_consensus.cc:374] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:33.355633 17577 heartbeater.cc:503] Master 127.0.98.62:40797 was elected leader, sending a full tablet report...
I20260928 17:55:33.355635 17862 raft_consensus.cc:729] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2258da473f40406bbc85f123848a6918, State: Initialized, Role: FOLLOWER
I20260928 17:55:33.355787 17862 consensus_queue.cc:260] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.356153 17866 tablet_bootstrap.cc:492] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: No bootstrap required, opened a new log
I20260928 17:55:33.356145 17845 heartbeater.cc:503] Master 127.0.98.62:40797 was elected leader, sending a full tablet report...
I20260928 17:55:33.356220 17866 ts_tablet_manager.cc:1397] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:33.356137 17862 ts_tablet_manager.cc:1428] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:33.358088 17866 raft_consensus.cc:348] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.358251 17866 raft_consensus.cc:374] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:33.358302 17866 raft_consensus.cc:729] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28bd1d12c6740a88c380baa3172ffa2, State: Initialized, Role: FOLLOWER
I20260928 17:55:33.358419 17866 consensus_queue.cc:260] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.358714 17866 ts_tablet_manager.cc:1428] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:33.358741 17711 heartbeater.cc:503] Master 127.0.98.62:40797 was elected leader, sending a full tablet report...
I20260928 17:55:33.361008 17869 raft_consensus.cc:482] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:33.361092 17869 raft_consensus.cc:504] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.361439 17869 leader_election.cc:288] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f28bd1d12c6740a88c380baa3172ffa2 (127.0.98.2:42461), 9c811981adc64e1aa89db56150947fa6 (127.0.98.1:32933)
I20260928 17:55:33.364454 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.361586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32933 (local address 127.0.98.3:58707)
0928 17:55:33.361707 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.361719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.361770 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.361948 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.361949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.361951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:33.362020 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.362022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.362661 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.362662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.363457 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.363460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.364237 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.364238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.364305 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.364310 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.364368 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.364401 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 17:55:33.364521 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.361657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58707 (local address 127.0.98.1:32933)
0928 17:55:33.361708 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.361711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.361842 (+ 131us) server_negotiation.cc:407] Connection header received
0928 17:55:33.361864 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.361865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.361874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.361932 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:33.362098 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.362575 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.363558 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.364150 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.364226 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.364235 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.364396 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.364430 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:55:33.364882 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.361595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42461 (local address 127.0.98.3:52381)
0928 17:55:33.361791 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.361805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.361894 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.362008 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.362009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.362010 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:33.362076 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.362078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.362695 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.362696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.363480 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.363482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.364588 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.364589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.364667 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.364670 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.364758 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.364798 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":37,"thread_start_us":97,"threads_started":1}
I20260928 17:55:33.364866 17531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e5da78194af6ad91d186937eded7" candidate_uuid: "2258da473f40406bbc85f123848a6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c811981adc64e1aa89db56150947fa6" is_pre_election: true
I20260928 17:55:33.365015 17531 raft_consensus.cc:2454] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2258da473f40406bbc85f123848a6918 in term 0.
I20260928 17:55:33.365290 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.361658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52381 (local address 127.0.98.2:42461)
0928 17:55:33.361744 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.361746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.361877 (+ 131us) server_negotiation.cc:407] Connection header received
0928 17:55:33.361952 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.361953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.361963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.362007 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:33.362151 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.362612 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.363587 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.364161 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.364226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.364236 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.365165 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.365218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 17:55:33.365286 17732 leader_election.cc:302] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2258da473f40406bbc85f123848a6918, 9c811981adc64e1aa89db56150947fa6; no voters:
I20260928 17:55:33.365443 17869 raft_consensus.cc:2781] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:33.365514 17869 raft_consensus.cc:482] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:33.365537 17869 raft_consensus.cc:3037] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:33.365562 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e5da78194af6ad91d186937eded7" candidate_uuid: "2258da473f40406bbc85f123848a6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28bd1d12c6740a88c380baa3172ffa2" is_pre_election: true
I20260928 17:55:33.365687 17665 raft_consensus.cc:2454] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2258da473f40406bbc85f123848a6918 in term 0.
I20260928 17:55:33.366338 17869 raft_consensus.cc:504] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.366478 17869 leader_election.cc:288] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [CANDIDATE]: Term 1 election: Requested vote from peers f28bd1d12c6740a88c380baa3172ffa2 (127.0.98.2:42461), 9c811981adc64e1aa89db56150947fa6 (127.0.98.1:32933)
I20260928 17:55:33.366624 17665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e5da78194af6ad91d186937eded7" candidate_uuid: "2258da473f40406bbc85f123848a6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28bd1d12c6740a88c380baa3172ffa2"
I20260928 17:55:33.366642 17531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b476e5da78194af6ad91d186937eded7" candidate_uuid: "2258da473f40406bbc85f123848a6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c811981adc64e1aa89db56150947fa6"
I20260928 17:55:33.366704 17665 raft_consensus.cc:3037] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:33.366706 17531 raft_consensus.cc:3037] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:33.367565 17531 raft_consensus.cc:2454] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2258da473f40406bbc85f123848a6918 in term 1.
I20260928 17:55:33.367565 17665 raft_consensus.cc:2454] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2258da473f40406bbc85f123848a6918 in term 1.
I20260928 17:55:33.367741 17734 leader_election.cc:302] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2258da473f40406bbc85f123848a6918, f28bd1d12c6740a88c380baa3172ffa2; no voters:
I20260928 17:55:33.367851 17869 raft_consensus.cc:2781] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:33.368016 17869 raft_consensus.cc:686] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 1 LEADER]: Becoming Leader. State: Replica: 2258da473f40406bbc85f123848a6918, State: Running, Role: LEADER
I20260928 17:55:33.368139 17869 consensus_queue.cc:237] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } }
I20260928 17:55:33.368973 17386 catalog_manager.cc:5495] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2258da473f40406bbc85f123848a6918 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2258da473f40406bbc85f123848a6918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2258da473f40406bbc85f123848a6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 } health_report { overall_health: UNKNOWN } } }
I20260928 17:55:33.373783 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.371562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.0.98.3:43085)
0928 17:55:33.371600 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.371602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.371625 (+ 23us) server_negotiation.cc:407] Connection header received
0928 17:55:33.371691 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.371692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.371705 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.371764 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:33.371927 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.372442 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.373190 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.373375 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.373425 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.373544 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:33.373627 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:33.373681 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.373724 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.373738 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:55:33.375581 17665 raft_consensus.cc:1261] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 1 FOLLOWER]: Refusing update from remote peer 2258da473f40406bbc85f123848a6918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:33.375656 17531 raft_consensus.cc:1261] T b476e5da78194af6ad91d186937eded7 P 9c811981adc64e1aa89db56150947fa6 [term 1 FOLLOWER]: Refusing update from remote peer 2258da473f40406bbc85f123848a6918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:33.375965 17872 consensus_queue.cc:1035] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c811981adc64e1aa89db56150947fa6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:33.376077 17869 consensus_queue.cc:1035] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bd1d12c6740a88c380baa3172ffa2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:33.378815 17876 mvcc.cc:205] Tried to move back new op lower bound from 7334371874303541248 to 7334371874276597760. Current Snapshot: MvccSnapshot[applied={T|T < 7334371874303541248}]
I20260928 17:55:33.379467 17877 mvcc.cc:205] Tried to move back new op lower bound from 7334371874303541248 to 7334371874276597760. Current Snapshot: MvccSnapshot[applied={T|T < 7334371874303541248}]
I20260928 17:55:33.381156 17386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:33.381258 17386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:33.382045 17386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e782f07d1245b1a8d53966b7557600: Sending DeleteTablet for 3 replicas of tablet b476e5da78194af6ad91d186937eded7
I20260928 17:55:33.382393 17511 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e5da78194af6ad91d186937eded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260928 17:55:33.382432 17645 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e5da78194af6ad91d186937eded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52650
I20260928 17:55:33.382663 17885 tablet_replica.cc:330] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: stopping tablet replica
I20260928 17:55:33.382812 17885 raft_consensus.cc:2229] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:33.382788 17779 tablet_service.cc:1514] Processing DeleteTablet for tablet b476e5da78194af6ad91d186937eded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45202
I20260928 17:55:33.382892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17448
I20260928 17:55:33.382988 17885 raft_consensus.cc:2258] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:33.388482 17885 ts_tablet_manager.cc:1905] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:33.388680 17886 tablet_replica.cc:330] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: stopping tablet replica
I20260928 17:55:33.388808 17886 raft_consensus.cc:2229] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:55:33.388914 17886 raft_consensus.cc:2258] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:33.389348 17886 ts_tablet_manager.cc:1905] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:33.390656 17885 ts_tablet_manager.cc:1918] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:33.390738 17885 log.cc:1196] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/b476e5da78194af6ad91d186937eded7
I20260928 17:55:33.391052 17885 ts_tablet_manager.cc:1939] T b476e5da78194af6ad91d186937eded7 P f28bd1d12c6740a88c380baa3172ffa2: Deleting consensus metadata
I20260928 17:55:33.391179 17886 ts_tablet_manager.cc:1918] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:33.391249 17886 log.cc:1196] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/b476e5da78194af6ad91d186937eded7
I20260928 17:55:33.391515 17886 ts_tablet_manager.cc:1939] T b476e5da78194af6ad91d186937eded7 P 2258da473f40406bbc85f123848a6918: Deleting consensus metadata
I20260928 17:55:33.391561 17372 catalog_manager.cc:4841] TS f28bd1d12c6740a88c380baa3172ffa2 (127.0.98.2:42461): tablet b476e5da78194af6ad91d186937eded7 (table test-table [id=d366c0a800544770b0c12c0cdf3f53da]) successfully deleted
I20260928 17:55:33.391920 17372 catalog_manager.cc:4841] TS 2258da473f40406bbc85f123848a6918 (127.0.98.3:43085): tablet b476e5da78194af6ad91d186937eded7 (table test-table [id=d366c0a800544770b0c12c0cdf3f53da]) successfully deleted
W20260928 17:55:33.392518 17370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32933 (error 108)
I20260928 17:55:33.393085 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.392952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32933 (local address 127.0.0.1:39996)
0928 17:55:33.393008 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.393046 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 17:55:33.393108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17582
W20260928 17:55:33.393744 17370 catalog_manager.cc:4584] TS 9c811981adc64e1aa89db56150947fa6 (127.0.98.1:32933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b476e5da78194af6ad91d186937eded7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32933: connect: Connection refused (error 111)
I20260928 17:55:33.400676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17716
I20260928 17:55:33.405557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17354
2026-09-28T17:55:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1900 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T17:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:33Z Disabled control of system clock
I20260928 17:55:33.433317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:33.515461 17891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.515702 17891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.515723 17891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.516927 17891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:33.516965 17891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.516978 17891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.516990 17891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:33.517004 17891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:33.518483 17891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.518700 17891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.518951 17891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:33.523927 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.523907 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.524000 17891 server_base.cc:1034] running on GCE node
W20260928 17:55:33.523907 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.524389 17891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.524675 17891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.525869 17891 hybrid_clock.cc:648] HybridClock initialized: now 1790618133525852 us; error 26 us; skew 500 ppm
I20260928 17:55:33.531836 17891 webserver.cc:460] Webserver started at http://127.0.98.62:40731/ using document root <none> and password file <none>
I20260928 17:55:33.532226 17891 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.532306 17891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.532505 17891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.533663 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "a5fedc7984664aeeb9ad6b773b24c1a8"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "51d2e36afe52d517ecc70d296e626e5a"
server_key_iv: "8040f3f160f9bf2752435174d4852031"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.534046 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5fedc7984664aeeb9ad6b773b24c1a8"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "51d2e36afe52d517ecc70d296e626e5a"
server_key_iv: "8040f3f160f9bf2752435174d4852031"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.535418 17891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:55:33.536280 17905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.536473 17891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:33.536583 17891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "a5fedc7984664aeeb9ad6b773b24c1a8"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "51d2e36afe52d517ecc70d296e626e5a"
server_key_iv: "8040f3f160f9bf2752435174d4852031"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.536660 17891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.557413 17891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.557825 17891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.557989 17891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.562275 17891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32893
I20260928 17:55:33.562322 17957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32893 every 8 connection(s)
I20260928 17:55:33.562691 17891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:33.563690 17958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:33.566228 17958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: Bootstrap starting.
I20260928 17:55:33.566834 17958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:33.567150 17958 log.cc:824] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:33.567873 17958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: No bootstrap required, opened a new log
I20260928 17:55:33.567945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17891
I20260928 17:55:33.568128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:33.568367 392 external_mini_cluster.cc:1428] Setting key 7bf8c940d478ff3dc6ed270344484470
I20260928 17:55:33.570281 17958 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260928 17:55:33.570466 17958 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:33.570504 17958 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fedc7984664aeeb9ad6b773b24c1a8, State: Initialized, Role: FOLLOWER
I20260928 17:55:33.570667 17958 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260928 17:55:33.570780 17958 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:33.570812 17958 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:33.570852 17958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:33.571615 17958 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260928 17:55:33.571727 17958 leader_election.cc:302] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5fedc7984664aeeb9ad6b773b24c1a8; no voters:
I20260928 17:55:33.571928 17958 leader_election.cc:288] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:33.571995 17963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:33.572198 17963 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [term 1 LEADER]: Becoming Leader. State: Replica: a5fedc7984664aeeb9ad6b773b24c1a8, State: Running, Role: LEADER
I20260928 17:55:33.572247 17958 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:33.572328 17963 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } }
I20260928 17:55:33.572743 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fedc7984664aeeb9ad6b773b24c1a8. Latest consensus state: current_term: 1 leader_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260928 17:55:33.572996 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:33.572924 17964 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fedc7984664aeeb9ad6b773b24c1a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32893 } } }
I20260928 17:55:33.573203 17972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:33.573225 17964 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:33.573840 17972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:33.575734 17972 catalog_manager.cc:1347] Generated new cluster ID: 4ead4cdfeed54d6b90f86cd2d273f15e
I20260928 17:55:33.575824 17972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:33.576501 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.569003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.98.62:32893)
0928 17:55:33.569234 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.569250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.569284 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:55:33.570231 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.570242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.570339 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.570498 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.570767 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.572671 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.573951 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.574594 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.574716 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.575598 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.575602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.575602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:33.575843 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.575890 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:33.575905 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.575973 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.576204 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.576297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260928 17:55:33.584375 17972 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:33.584949 17972 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:33.589730 17972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: Generated new TSK 0
I20260928 17:55:33.589913 17972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:33.597306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--builtin_ntp_servers=127.0.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:33.689459 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.689673 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.689695 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.690891 17982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.690934 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.690961 17982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:33.692435 17982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.692667 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.692978 17982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:33.698596 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.698689 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.698626 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.699111 17982 server_base.cc:1034] running on GCE node
I20260928 17:55:33.699335 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.699572 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.700742 17982 hybrid_clock.cc:648] HybridClock initialized: now 1790618133700716 us; error 37 us; skew 500 ppm
I20260928 17:55:33.702029 17982 webserver.cc:460] Webserver started at http://127.0.98.1:41093/ using document root <none> and password file <none>
I20260928 17:55:33.702337 17982 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.702420 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.702562 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.703649 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "9ee367ae0a2b452f169f8a093f95af1c"
server_key_iv: "8d9e0377075510819c521f2aebe3dc5d"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.704018 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "9ee367ae0a2b452f169f8a093f95af1c"
server_key_iv: "8d9e0377075510819c521f2aebe3dc5d"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.705478 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:33.706352 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.706599 17982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:33.706697 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "9ee367ae0a2b452f169f8a093f95af1c"
server_key_iv: "8d9e0377075510819c521f2aebe3dc5d"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.706789 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.726318 17982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.726683 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.726833 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.727095 17982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:33.727450 17982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:33.727487 17982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.727512 17982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:33.727525 17982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.733330 17982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37213
I20260928 17:55:33.733426 18109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37213 every 8 connection(s)
I20260928 17:55:33.733752 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:55:33.739120 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.734530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53443 (local address 127.0.98.62:32893)
0928 17:55:33.734593 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.734595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.734852 (+ 257us) server_negotiation.cc:407] Connection header received
0928 17:55:33.735404 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.735406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.735422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.735473 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.736210 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.736715 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.737698 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.737904 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.737985 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.738815 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.738817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.738817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:33.738821 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.738836 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:33.738841 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.738909 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.739006 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.739038 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 17:55:33.739337 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.734515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.1:53443)
0928 17:55:33.734722 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.734761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.735288 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.735549 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.735553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.735561 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:33.736067 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.736071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.736813 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.736814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.737601 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.737603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.737992 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.737993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.738130 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.738630 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:33.738631 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:33.738659 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:33.738681 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:33.738910 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:33.738917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:33.738920 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.739009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.739110 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260928 17:55:33.739866 18110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260928 17:55:33.739948 18110 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:33.740151 18110 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260928 17:55:33.740679 17922 ts_manager.cc:194] Registered new tserver with Master: a6f6f9bb16e347d4b403b54ddbd42a2f (127.0.98.1:37213)
I20260928 17:55:33.741303 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53443
I20260928 17:55:33.743006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 17982
I20260928 17:55:33.743089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:33.743254 392 external_mini_cluster.cc:1428] Setting key b4c94d8420016f053cb5a02315bf8536
I20260928 17:55:33.744233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--builtin_ntp_servers=127.0.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:33.823992 18113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.824201 18113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.824223 18113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.825384 18113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.825435 18113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.825461 18113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:33.826778 18113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.826951 18113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.827164 18113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:33.831969 18121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.831945 18118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.832056 18113 server_base.cc:1034] running on GCE node
W20260928 17:55:33.831954 18119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.832403 18113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.832650 18113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.833791 18113 hybrid_clock.cc:648] HybridClock initialized: now 1790618133833779 us; error 27 us; skew 500 ppm
I20260928 17:55:33.834975 18113 webserver.cc:460] Webserver started at http://127.0.98.2:40459/ using document root <none> and password file <none>
I20260928 17:55:33.835160 18113 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.835201 18113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.835286 18113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.836345 18113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b6c6465905643ec99e0535e5d892daa"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "a107f16219231f480cc87ec250ff2c22"
server_key_iv: "2e0d1e9c68b987b0c66ee726be2349fb"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.836647 18113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b6c6465905643ec99e0535e5d892daa"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "a107f16219231f480cc87ec250ff2c22"
server_key_iv: "2e0d1e9c68b987b0c66ee726be2349fb"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.837930 18113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:33.838647 18127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.838858 18113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:33.838953 18113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "2b6c6465905643ec99e0535e5d892daa"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "a107f16219231f480cc87ec250ff2c22"
server_key_iv: "2e0d1e9c68b987b0c66ee726be2349fb"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.839025 18113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.849249 18113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.849503 18113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.849602 18113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.849781 18113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:33.850049 18113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:33.850080 18113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.850100 18113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:33.850113 18113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.855958 18113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33801
I20260928 17:55:33.855997 18240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33801 every 8 connection(s)
I20260928 17:55:33.856392 18113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:55:33.859180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18113
I20260928 17:55:33.859298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:33.859543 392 external_mini_cluster.cc:1428] Setting key 8b2ddb483309356226e254e87ad50608
I20260928 17:55:33.860670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--builtin_ntp_servers=127.0.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 17:55:33.862406 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.857163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49049 (local address 127.0.98.62:32893)
0928 17:55:33.857245 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.857247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.857529 (+ 282us) server_negotiation.cc:407] Connection header received
0928 17:55:33.858017 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.858018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.858029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.858097 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.858783 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.859279 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.860791 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.861045 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.861103 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.862059 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.862061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.862061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:33.862067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.862088 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:33.862094 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.862158 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.862263 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.862293 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 17:55:33.862601 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.857145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.2:49049)
0928 17:55:33.857379 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.857421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.857930 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.858177 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.858181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.858189 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:33.858701 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.858705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.859408 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.859409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.860697 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.860700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.861139 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.861140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.861321 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.861877 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:33.861878 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:33.861921 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:33.861954 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:33.862168 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:33.862173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:33.862176 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.862243 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.862334 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260928 17:55:33.863317 18241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260928 17:55:33.863427 18241 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:33.863610 18241 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260928 17:55:33.864042 17922 ts_manager.cc:194] Registered new tserver with Master: 2b6c6465905643ec99e0535e5d892daa (127.0.98.2:33801)
I20260928 17:55:33.864462 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49049
W20260928 17:55:33.944154 18244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:33.944373 18244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:33.944409 18244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:33.945585 18244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:33.945658 18244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:33.945708 18244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:33.947016 18244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:33.947212 18244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:33.947460 18244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:55:33.952805 18244 server_base.cc:1034] running on GCE node
W20260928 17:55:33.952911 18249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.952945 18250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:33.953037 18252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:33.953280 18244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:33.953562 18244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:33.954730 18244 hybrid_clock.cc:648] HybridClock initialized: now 1790618133954719 us; error 45 us; skew 500 ppm
I20260928 17:55:33.956197 18244 webserver.cc:460] Webserver started at http://127.0.98.3:44457/ using document root <none> and password file <none>
I20260928 17:55:33.956463 18244 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:33.956533 18244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:33.956709 18244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:33.957793 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "801ed06427664c8c8383d12c7b1d4b04"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "3fced02be20ec9c351001b06fbf0ffd2"
server_key_iv: "f1a598f495356e6d5d2016578fcc8be6"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.958176 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "801ed06427664c8c8383d12c7b1d4b04"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "3fced02be20ec9c351001b06fbf0ffd2"
server_key_iv: "f1a598f495356e6d5d2016578fcc8be6"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.959502 18244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:33.960285 18258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.960469 18244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:33.960565 18244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "801ed06427664c8c8383d12c7b1d4b04"
format_stamp: "Formatted at 2026-09-28 17:55:33 on dist-test-slave-zqlt"
server_key: "3fced02be20ec9c351001b06fbf0ffd2"
server_key_iv: "f1a598f495356e6d5d2016578fcc8be6"
server_key_version: "encryptionkey@0"
I20260928 17:55:33.960646 18244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:33.977764 18244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:33.978117 18244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:33.978282 18244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:33.978523 18244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:33.978871 18244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:33.978925 18244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.978960 18244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:33.978987 18244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:33.984825 18244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39157
I20260928 17:55:33.984862 18371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39157 every 8 connection(s)
I20260928 17:55:33.985186 18244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:55:33.986357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18244
I20260928 17:55:33.986465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:33.986703 392 external_mini_cluster.cc:1428] Setting key 15e4fa01c824e3e97b2a312cd1dad5f8
I20260928 17:55:33.990725 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.986050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36171 (local address 127.0.98.62:32893)
0928 17:55:33.986139 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:33.986142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:33.986456 (+ 314us) server_negotiation.cc:407] Connection header received
0928 17:55:33.987175 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:33.987176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:33.987189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:33.987266 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:33.988044 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.988600 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.989540 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.989711 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.989761 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.990501 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:33.990502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:33.990503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:33.990506 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:33.990520 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:33.990524 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:33.990567 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:33.990646 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:55:33.990668 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:55:33.990971 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:33.986029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32893 (local address 127.0.98.3:36171)
0928 17:55:33.986295 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:33.986348 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:33.987058 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:33.987331 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:33.987335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:33.987347 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:33.987965 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.987970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.988681 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.988682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.989470 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:33.989472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:33.989783 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:33.989784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:33.989887 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:33.990376 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:33.990376 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:33.990404 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:33.990426 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:33.990579 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:33.990583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:33.990585 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:55:33.990636 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:55:33.990710 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260928 17:55:33.991464 18372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32893
I20260928 17:55:33.991544 18372 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:33.991770 18372 heartbeater.cc:511] Master 127.0.98.62:32893 requested a full tablet report, sending...
I20260928 17:55:33.992192 17922 ts_manager.cc:194] Registered new tserver with Master: 801ed06427664c8c8383d12c7b1d4b04 (127.0.98.3:39157)
I20260928 17:55:33.992551 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36171
I20260928 17:55:33.998649 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:34.003157 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.000147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.0.98.62:32893)
0928 17:55:34.000207 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.000209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.000288 (+ 79us) server_negotiation.cc:407] Connection header received
0928 17:55:34.000365 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.000366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.000380 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.000449 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:34.000630 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.001112 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.001885 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.002078 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.002154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.002871 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:34.002873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:34.002873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:34.002879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:34.002894 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:34.002903 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:34.002958 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.003031 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.003058 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:55:34.004643 17922 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:34.006313 17922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:34.013492 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.008665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.0.1:49704)
0928 17:55:34.008865 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.008874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:34.008914 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:34.009354 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:34.009357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:34.009361 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:34.009457 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.009460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010896 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.010897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.012282 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.012284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.013180 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.013182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.013264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.013279 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 17:55:34.013355 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 17:55:34.013404 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":69,"thread_start_us":110,"threads_started":1}
I20260928 17:55:34.013504 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.008937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.0.98.3:39157)
0928 17:55:34.009145 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.009151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.009163 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:55:34.009223 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.009225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.009240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.009538 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:34.009603 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010772 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.012353 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.013094 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.013173 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.013270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.013379 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.013411 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260928 17:55:34.014096 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.008665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33801 (local address 127.0.0.1:51792)
0928 17:55:34.009465 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.009475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:34.009528 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:34.009778 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:34.009778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:34.009780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:34.009832 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.009834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010785 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.010786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.012910 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.012911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.013853 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.013854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.013924 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.013928 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:34.013966 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 17:55:34.014022 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":72,"thread_start_us":109,"threads_started":1}
I20260928 17:55:34.014118 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.009146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.98.2:33801)
0928 17:55:34.009404 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.009409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.009584 (+ 175us) server_negotiation.cc:407] Connection header received
0928 17:55:34.009634 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.009636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.009648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.009732 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:34.009915 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010696 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.012996 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.013778 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.013862 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.013957 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.013993 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.014034 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":133,"threads_started":1}
I20260928 17:55:34.014285 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.008633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37213 (local address 127.0.0.1:54862)
0928 17:55:34.008765 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.008807 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:34.008879 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:34.009354 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:34.009357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:34.009360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:34.009457 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.009460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010858 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.010859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.012213 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.012216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.014030 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.014031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.014100 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.014104 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:34.014158 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:55:34.014208 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
I20260928 17:55:34.014593 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.008959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.0.98.1:37213)
0928 17:55:34.009150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.009153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.009163 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:55:34.009205 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.009207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.009225 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.009537 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:34.009603 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.010777 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.012336 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.013094 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.013173 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.013270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.014459 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.014507 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260928 17:55:34.014510 18306 tablet_service.cc:1467] Processing CreateTablet for tablet 018ddfd751f4441d95aae8f694ff00ac (DEFAULT_TABLE table=test-table [id=d502c80f4af34b6c9c6ff896bbd25453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:34.014940 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018ddfd751f4441d95aae8f694ff00ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:34.014964 18175 tablet_service.cc:1467] Processing CreateTablet for tablet 018ddfd751f4441d95aae8f694ff00ac (DEFAULT_TABLE table=test-table [id=d502c80f4af34b6c9c6ff896bbd25453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:34.015303 18175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018ddfd751f4441d95aae8f694ff00ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:34.015290 18044 tablet_service.cc:1467] Processing CreateTablet for tablet 018ddfd751f4441d95aae8f694ff00ac (DEFAULT_TABLE table=test-table [id=d502c80f4af34b6c9c6ff896bbd25453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:34.015617 18044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018ddfd751f4441d95aae8f694ff00ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:34.018335 18386 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Bootstrap starting.
I20260928 17:55:34.018888 18387 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Bootstrap starting.
I20260928 17:55:34.019032 18386 tablet_bootstrap.cc:654] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:34.019335 18386 log.cc:824] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:34.019512 18387 tablet_bootstrap.cc:654] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:34.019835 18387 log.cc:824] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:34.020201 18386 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: No bootstrap required, opened a new log
I20260928 17:55:34.020200 18388 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Bootstrap starting.
I20260928 17:55:34.020275 18386 ts_tablet_manager.cc:1397] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:34.020733 18388 tablet_bootstrap.cc:654] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:34.021016 18388 log.cc:824] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:34.021457 18387 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: No bootstrap required, opened a new log
I20260928 17:55:34.021548 18387 ts_tablet_manager.cc:1397] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 17:55:34.021795 18388 tablet_bootstrap.cc:492] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: No bootstrap required, opened a new log
I20260928 17:55:34.021862 18388 ts_tablet_manager.cc:1397] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 17:55:34.022362 18386 raft_consensus.cc:348] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.022517 18386 raft_consensus.cc:374] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:34.022559 18386 raft_consensus.cc:729] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f6f9bb16e347d4b403b54ddbd42a2f, State: Initialized, Role: FOLLOWER
I20260928 17:55:34.022673 18386 consensus_queue.cc:260] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.022941 18386 ts_tablet_manager.cc:1428] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:34.023072 18110 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
I20260928 17:55:34.024140 18387 raft_consensus.cc:348] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.024328 18387 raft_consensus.cc:374] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:34.024369 18387 raft_consensus.cc:729] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801ed06427664c8c8383d12c7b1d4b04, State: Initialized, Role: FOLLOWER
I20260928 17:55:34.024482 18387 consensus_queue.cc:260] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.024559 18388 raft_consensus.cc:348] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.024708 18388 raft_consensus.cc:374] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:34.024807 18387 ts_tablet_manager.cc:1428] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:34.024789 18388 raft_consensus.cc:729] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6c6465905643ec99e0535e5d892daa, State: Initialized, Role: FOLLOWER
I20260928 17:55:34.024860 18372 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
I20260928 17:55:34.024920 18388 consensus_queue.cc:260] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.025174 18388 ts_tablet_manager.cc:1428] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 17:55:34.025206 18241 heartbeater.cc:503] Master 127.0.98.62:32893 was elected leader, sending a full tablet report...
W20260928 17:55:34.106928 18242 tablet.cc:2369] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:55:34.234421 18111 tablet.cc:2369] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 17:55:34.235692 18373 tablet.cc:2369] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:55:34.258438 18392 raft_consensus.cc:482] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:34.258566 18392 raft_consensus.cc:504] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.259035 18392 leader_election.cc:288] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6c6465905643ec99e0535e5d892daa (127.0.98.2:33801), 801ed06427664c8c8383d12c7b1d4b04 (127.0.98.3:39157)
I20260928 17:55:34.262210 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.259362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53315 (local address 127.0.98.3:39157)
0928 17:55:34.259413 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.259416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.259524 (+ 108us) server_negotiation.cc:407] Connection header received
0928 17:55:34.259621 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.259623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.259632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.259678 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:34.259878 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.260490 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261491 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.261856 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261937 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.261946 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.262098 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.262138 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:55:34.262204 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.259311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42883 (local address 127.0.98.2:33801)
0928 17:55:34.259378 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.259382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.259499 (+ 117us) server_negotiation.cc:407] Connection header received
0928 17:55:34.259600 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.259601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.259611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.259679 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:34.259887 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.260468 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261489 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.261852 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261937 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.261946 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.262100 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.262135 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:55:34.262238 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.259250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39157 (local address 127.0.98.1:53315)
0928 17:55:34.259438 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.259448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:34.259553 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:34.259679 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:34.259680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:34.259681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:34.259758 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.259760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.260578 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.260579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.261396 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.261397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.261939 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.262014 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.262018 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:34.262093 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:55:34.262134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260928 17:55:34.262243 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.259239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33801 (local address 127.0.98.1:42883)
0928 17:55:34.259411 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.259422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:34.259513 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:34.259696 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:34.259697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:34.259699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:34.259809 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.259811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.260564 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.260566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.261393 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:34.261395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.261935 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.261936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:34.262007 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.262011 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:34.262083 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 17:55:34.262125 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260928 17:55:34.262612 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018ddfd751f4441d95aae8f694ff00ac" candidate_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801ed06427664c8c8383d12c7b1d4b04" is_pre_election: true
I20260928 17:55:34.262802 18326 raft_consensus.cc:2454] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f6f9bb16e347d4b403b54ddbd42a2f in term 0.
I20260928 17:55:34.262773 18195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018ddfd751f4441d95aae8f694ff00ac" candidate_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6c6465905643ec99e0535e5d892daa" is_pre_election: true
I20260928 17:55:34.262907 18195 raft_consensus.cc:2454] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f6f9bb16e347d4b403b54ddbd42a2f in term 0.
I20260928 17:55:34.263108 17998 leader_election.cc:302] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801ed06427664c8c8383d12c7b1d4b04, a6f6f9bb16e347d4b403b54ddbd42a2f; no voters:
I20260928 17:55:34.263262 18392 raft_consensus.cc:2781] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:34.263329 18392 raft_consensus.cc:482] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:34.263347 18392 raft_consensus.cc:3037] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:34.264032 18392 raft_consensus.cc:504] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.264252 18392 leader_election.cc:288] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [CANDIDATE]: Term 1 election: Requested vote from peers 2b6c6465905643ec99e0535e5d892daa (127.0.98.2:33801), 801ed06427664c8c8383d12c7b1d4b04 (127.0.98.3:39157)
I20260928 17:55:34.264403 18195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018ddfd751f4441d95aae8f694ff00ac" candidate_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6c6465905643ec99e0535e5d892daa"
I20260928 17:55:34.264470 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018ddfd751f4441d95aae8f694ff00ac" candidate_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801ed06427664c8c8383d12c7b1d4b04"
I20260928 17:55:34.264515 18195 raft_consensus.cc:3037] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:34.264580 18326 raft_consensus.cc:3037] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:34.265393 18326 raft_consensus.cc:2454] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f6f9bb16e347d4b403b54ddbd42a2f in term 1.
I20260928 17:55:34.265403 18195 raft_consensus.cc:2454] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f6f9bb16e347d4b403b54ddbd42a2f in term 1.
I20260928 17:55:34.265642 17998 leader_election.cc:302] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801ed06427664c8c8383d12c7b1d4b04, a6f6f9bb16e347d4b403b54ddbd42a2f; no voters:
I20260928 17:55:34.265748 18392 raft_consensus.cc:2781] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:34.265911 18392 raft_consensus.cc:686] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 1 LEADER]: Becoming Leader. State: Replica: a6f6f9bb16e347d4b403b54ddbd42a2f, State: Running, Role: LEADER
I20260928 17:55:34.266027 18392 consensus_queue.cc:237] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } }
I20260928 17:55:34.267064 17922 catalog_manager.cc:5495] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f6f9bb16e347d4b403b54ddbd42a2f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f6f9bb16e347d4b403b54ddbd42a2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 } health_report { overall_health: UNKNOWN } } }
I20260928 17:55:34.295125 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.292813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.0.98.1:37213)
0928 17:55:34.292875 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:34.292877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:34.292884 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:55:34.292922 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:34.292923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:34.292945 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:34.293009 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:34.293134 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.293608 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.294347 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:34.294527 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:34.294578 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:34.294706 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:34.294884 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:34.294966 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:34.294986 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 17:55:34.295004 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:55:34.297150 18326 raft_consensus.cc:1261] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 1 FOLLOWER]: Refusing update from remote peer a6f6f9bb16e347d4b403b54ddbd42a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:34.297159 18195 raft_consensus.cc:1261] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 1 FOLLOWER]: Refusing update from remote peer a6f6f9bb16e347d4b403b54ddbd42a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:34.297569 18397 consensus_queue.cc:1035] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "801ed06427664c8c8383d12c7b1d4b04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:34.297724 18392 consensus_queue.cc:1035] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6c6465905643ec99e0535e5d892daa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:34.300123 18400 mvcc.cc:205] Tried to move back new op lower bound from 7334371878078218240 to 7334371877954674688. Current Snapshot: MvccSnapshot[applied={T|T < 7334371878078218240}]
I20260928 17:55:34.300441 18401 mvcc.cc:205] Tried to move back new op lower bound from 7334371878078218240 to 7334371877954674688. Current Snapshot: MvccSnapshot[applied={T|T < 7334371878078218240}]
I20260928 17:55:34.302498 17922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:34.302584 17922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:34.303349 17922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5fedc7984664aeeb9ad6b773b24c1a8: Sending DeleteTablet for 3 replicas of tablet 018ddfd751f4441d95aae8f694ff00ac
I20260928 17:55:34.303654 18044 tablet_service.cc:1514] Processing DeleteTablet for tablet 018ddfd751f4441d95aae8f694ff00ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:34 UTC) from {username='slave'} at 127.0.0.1:54862
I20260928 17:55:34.303685 18175 tablet_service.cc:1514] Processing DeleteTablet for tablet 018ddfd751f4441d95aae8f694ff00ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:34 UTC) from {username='slave'} at 127.0.0.1:51792
I20260928 17:55:34.303876 18409 tablet_replica.cc:330] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f: stopping tablet replica
I20260928 17:55:34.303974 18409 raft_consensus.cc:2229] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:55:34.304029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17982
I20260928 17:55:34.304069 18409 raft_consensus.cc:2258] T 018ddfd751f4441d95aae8f694ff00ac P a6f6f9bb16e347d4b403b54ddbd42a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:34.304160 18306 tablet_service.cc:1514] Processing DeleteTablet for tablet 018ddfd751f4441d95aae8f694ff00ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:34 UTC) from {username='slave'} at 127.0.0.1:49704
I20260928 17:55:34.305459 18411 tablet_replica.cc:330] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: stopping tablet replica
I20260928 17:55:34.305552 18411 raft_consensus.cc:2229] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:34.305648 18411 raft_consensus.cc:2258] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:34.306090 18411 ts_tablet_manager.cc:1905] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:34.308174 18411 ts_tablet_manager.cc:1918] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:34.308254 18411 log.cc:1196] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/018ddfd751f4441d95aae8f694ff00ac
I20260928 17:55:34.308219 18410 tablet_replica.cc:330] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: stopping tablet replica
I20260928 17:55:34.308344 18410 raft_consensus.cc:2229] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:34.308444 18410 raft_consensus.cc:2258] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:34.308498 18411 ts_tablet_manager.cc:1939] T 018ddfd751f4441d95aae8f694ff00ac P 801ed06427664c8c8383d12c7b1d4b04: Deleting consensus metadata
I20260928 17:55:34.308887 18410 ts_tablet_manager.cc:1905] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:34.309042 17907 catalog_manager.cc:4841] TS 801ed06427664c8c8383d12c7b1d4b04 (127.0.98.3:39157): tablet 018ddfd751f4441d95aae8f694ff00ac (table test-table [id=d502c80f4af34b6c9c6ff896bbd25453]) successfully deleted
I20260928 17:55:34.310307 18410 ts_tablet_manager.cc:1918] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:34.310382 18410 log.cc:1196] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/018ddfd751f4441d95aae8f694ff00ac
W20260928 17:55:34.310494 17906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37213 (error 108)
I20260928 17:55:34.310655 18410 ts_tablet_manager.cc:1939] T 018ddfd751f4441d95aae8f694ff00ac P 2b6c6465905643ec99e0535e5d892daa: Deleting consensus metadata
I20260928 17:55:34.310702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18113
I20260928 17:55:34.311736 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.311576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37213 (local address 127.0.0.1:54872)
0928 17:55:34.311643 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.311695 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 17:55:34.311894 17906 catalog_manager.cc:4584] TS a6f6f9bb16e347d4b403b54ddbd42a2f (127.0.98.1:37213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 018ddfd751f4441d95aae8f694ff00ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37213: connect: Connection refused (error 111)
I20260928 17:55:34.316138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18244
I20260928 17:55:34.316171 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:34.315998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33801 (local address 127.0.0.1:51798)
0928 17:55:34.316069 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:34.316127 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 17:55:34.320777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 17891
2026-09-28T17:55:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (914 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:34 dist-test-slave-zqlt krb5kdc[18415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:34 dist-test-slave-zqlt krb5kdc[18415](info): set up 2 sockets
Sep 28 17:55:34 dist-test-slave-zqlt krb5kdc[18415](info): commencing operation
krb5kdc: starting...
W20260928 17:55:35.359913 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:35.478468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:35.559603 18428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:35.559839 18428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:35.559862 18428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:35.561117 18428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:35.561164 18428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:35.561179 18428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:35.561192 18428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:35.561203 18428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:35.562820 18428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:35.563079 18428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:35.563345 18428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:35.568404 18436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:35.568452 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:35.568645 18428 server_base.cc:1034] running on GCE node
W20260928 17:55:35.568405 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:35.568938 18428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:35.569187 18428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:35.570380 18428 hybrid_clock.cc:648] HybridClock initialized: now 1790618135570365 us; error 26 us; skew 500 ppm
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:35.572914 18428 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:35.573390 18428 webserver.cc:460] Webserver started at http://127.0.98.62:32919/ using document root <none> and password file <none>
I20260928 17:55:35.573625 18428 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:35.573671 18428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:35.573840 18428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:35.574915 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "37aa0d03ad587d1b06da4964db8da617"
server_key_iv: "6457baeade037ab5c2b81d21a14514a7"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.575274 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "37aa0d03ad587d1b06da4964db8da617"
server_key_iv: "6457baeade037ab5c2b81d21a14514a7"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.576654 18428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:35.577386 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.577622 18428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:35.577767 18428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "37aa0d03ad587d1b06da4964db8da617"
server_key_iv: "6457baeade037ab5c2b81d21a14514a7"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.577841 18428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:35.602713 18428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:35.604697 18428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:35.604857 18428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:35.608843 18428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41497
I20260928 17:55:35.608897 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41497 every 8 connection(s)
I20260928 17:55:35.609242 18428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:35.610132 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:35.612569 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: Bootstrap starting.
I20260928 17:55:35.613121 18496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:35.613363 18496 log.cc:824] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:35.613442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18428
I20260928 17:55:35.613574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:35.613757 392 external_mini_cluster.cc:1428] Setting key 1d802729877257312cf0634ef1a78c3d
I20260928 17:55:35.614151 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: No bootstrap required, opened a new log
I20260928 17:55:35.615964 18496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260928 17:55:35.616146 18496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:35.616211 18496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6cc0f6f7374d16ac79c4df47e12bdc, State: Initialized, Role: FOLLOWER
I20260928 17:55:35.616372 18496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260928 17:55:35.616470 18496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:35.616515 18496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:35.616567 18496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:35.617368 18496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260928 17:55:35.617483 18496 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6cc0f6f7374d16ac79c4df47e12bdc; no voters:
I20260928 17:55:35.617684 18496 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:35.617703 18501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:35.617862 18501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [term 1 LEADER]: Becoming Leader. State: Replica: 2e6cc0f6f7374d16ac79c4df47e12bdc, State: Running, Role: LEADER
I20260928 17:55:35.617996 18496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:35.618000 18501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260928 17:55:35.618423 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260928 17:55:35.618434 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6cc0f6f7374d16ac79c4df47e12bdc. Latest consensus state: current_term: 1 leader_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6cc0f6f7374d16ac79c4df47e12bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260928 17:55:35.618520 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:35.618553 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:35.618734 18507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:35.619330 18507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:35.622047 18507 catalog_manager.cc:1347] Generated new cluster ID: 7796824fbb6449e380474b0d7b0c171a
I20260928 17:55:35.622104 18507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:35.626211 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:35.614396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.0.98.62:41497)
0928 17:55:35.614605 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:35.614614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:35.614649 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:55:35.615954 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:35.615962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:35.616016 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:35.616233 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:35.616974 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.618113 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.619221 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.619858 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.619986 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:35.622788 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:35.622792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:35.622793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:35.622805 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:35.624591 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.624987 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.624991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.624991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.625112 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.625331 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.625332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.625333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.625584 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:35.625749 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:35.625848 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:55:35.625997 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260928 17:55:35.637693 18507 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:35.638443 18507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:35.641846 18507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: Generated new TSK 0
I20260928 17:55:35.642050 18507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:55:35.679066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:35.759968 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:35.760244 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:35.760290 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:35.761463 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:35.761528 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:35.761572 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:35.763111 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:35.763372 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:35.763598 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:35.768637 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:35.768637 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:35.768637 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:35.768913 18522 server_base.cc:1034] running on GCE node
I20260928 17:55:35.769212 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:35.769484 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:35.770656 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790618135770644 us; error 30 us; skew 500 ppm
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:35.773206 18522 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:55:35.773635 18522 webserver.cc:460] Webserver started at http://127.0.98.1:36525/ using document root <none> and password file <none>
I20260928 17:55:35.773919 18522 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:35.773988 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:35.774145 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:35.775202 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "12010470bcde4ab5a5293d24b9d8a64b"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "663fea3eeb0e32421ee904b064f45d8c"
server_key_iv: "7f912ec2815f4748c8e31cbd957f77cd"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.775565 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12010470bcde4ab5a5293d24b9d8a64b"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "663fea3eeb0e32421ee904b064f45d8c"
server_key_iv: "7f912ec2815f4748c8e31cbd957f77cd"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.776958 18522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:55:35.777693 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.777894 18522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:35.777992 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "12010470bcde4ab5a5293d24b9d8a64b"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "663fea3eeb0e32421ee904b064f45d8c"
server_key_iv: "7f912ec2815f4748c8e31cbd957f77cd"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.778069 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:35.806134 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:35.808492 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:35.808684 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:35.808948 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:35.809280 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:35.809341 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.809386 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:35.809430 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.815307 18522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45423
I20260928 17:55:35.815359 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45423 every 8 connection(s)
I20260928 17:55:35.815685 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:35.824607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18522
I20260928 17:55:35.824713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:35.824950 392 external_mini_cluster.cc:1428] Setting key 4c15c014c124186834c32e9a4ede77a6
I20260928 17:55:35.826043 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:35.816462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36829 (local address 127.0.98.62:41497)
0928 17:55:35.816508 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:35.816510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:35.816760 (+ 250us) server_negotiation.cc:407] Connection header received
0928 17:55:35.817750 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:35.817751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:35.817768 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:35.817833 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:35.818579 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.819251 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.820432 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.820656 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.820742 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:35.822840 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:35.822841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:35.822841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:35.822850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:35.824355 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.824870 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.824872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.824872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.824925 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.825285 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.825286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.825287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.825450 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:35.825516 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:35.825854 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 17:55:35.825945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:55:35.826251 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:35.816429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.1:36829)
0928 17:55:35.816618 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:35.816679 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:35.817616 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:35.817913 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:35.817917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:35.818276 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:35.818480 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:35.818486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.819345 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.819346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:35.820333 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:35.820336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.820751 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.820752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:35.820881 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:35.821363 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:35.821370 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:35.822743 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:35.824473 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:35.824479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:35.824480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:35.824769 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:35.825047 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:35.825048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:35.825049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:35.825200 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:35.825535 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:35.825541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:35.825710 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 17:55:35.825852 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 17:55:35.826012 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260928 17:55:35.826764 18651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260928 17:55:35.826856 18651 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:35.827047 18651 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260928 17:55:35.827615 18460 ts_manager.cc:194] Registered new tserver with Master: 12010470bcde4ab5a5293d24b9d8a64b (127.0.98.1:45423)
I20260928 17:55:35.828292 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36829
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:55:35.850332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:35.930382 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:35.930619 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:35.930655 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:35.931777 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:35.931846 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:35.931890 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:35.933419 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:35.933676 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:35.933900 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:35.938599 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:35.938601 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:35.938663 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:35.938838 18656 server_base.cc:1034] running on GCE node
I20260928 17:55:35.939039 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:35.939285 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:35.940448 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790618135940442 us; error 42 us; skew 500 ppm
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:35.942827 18656 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:55:35.943287 18656 webserver.cc:460] Webserver started at http://127.0.98.2:33911/ using document root <none> and password file <none>
I20260928 17:55:35.943516 18656 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:35.943584 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:35.943735 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:35.944830 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "43c9186bf4fd4eaca1c5c8a8cc669334"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "3645e763959f813ee3b6ab11a9723ae2"
server_key_iv: "88daf72796d02bbd0842d75d5749e3a3"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.945176 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43c9186bf4fd4eaca1c5c8a8cc669334"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "3645e763959f813ee3b6ab11a9723ae2"
server_key_iv: "88daf72796d02bbd0842d75d5749e3a3"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.946496 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:35.947278 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.947489 18656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:35.947587 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "43c9186bf4fd4eaca1c5c8a8cc669334"
format_stamp: "Formatted at 2026-09-28 17:55:35 on dist-test-slave-zqlt"
server_key: "3645e763959f813ee3b6ab11a9723ae2"
server_key_iv: "88daf72796d02bbd0842d75d5749e3a3"
server_key_version: "encryptionkey@0"
I20260928 17:55:35.947669 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:35.966185 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:35.968546 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:35.968705 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:35.968972 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:35.969326 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:35.969388 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.969430 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:35.969457 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:35.975414 18656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40929
I20260928 17:55:35.975507 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40929 every 8 connection(s)
I20260928 17:55:35.975854 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:55:35 dist-test-slave-zqlt krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:35.985405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18656
I20260928 17:55:35.985489 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:35.976668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55785 (local address 127.0.98.62:41497)
0928 17:55:35.976724 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:35.976725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:35.976956 (+ 231us) server_negotiation.cc:407] Connection header received
0928 17:55:35.977944 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:35.977946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:35.977959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:35.978009 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:35.978698 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.979226 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.980417 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.980657 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.980730 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:35.982922 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:35.982923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:35.982923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:35.982930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:35.984210 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.984569 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.984569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.984570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.984620 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:35.984826 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:35.984826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:35.984827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:35.984934 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:35.984987 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:35.985250 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 17:55:35.985367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 17:55:35.985536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:35.985522 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:35.976638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.2:55785)
0928 17:55:35.976822 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:35.976864 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:35.977818 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:35.978090 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:35.978094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:35.978366 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:35.978597 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:35.978601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.979331 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.979332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:35.980310 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:35.980313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:35.980738 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:35.980739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:35.980872 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:35.981418 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:35.981426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:35.982833 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:35.984281 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:35.984286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:35.984288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:35.984505 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:35.984676 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:35.984676 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:35.984677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:35.984771 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:35.984990 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:35.984994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:35.985109 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 17:55:35.985193 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:55:35.985311 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260928 17:55:35.985769 392 external_mini_cluster.cc:1428] Setting key 1c6fcd49bfb5ab14c99c813b835810c8
I20260928 17:55:35.986290 18785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260928 17:55:35.986392 18785 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:35.986572 18785 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260928 17:55:35.986996 18460 ts_manager.cc:194] Registered new tserver with Master: 43c9186bf4fd4eaca1c5c8a8cc669334 (127.0.98.2:40929)
I20260928 17:55:35.987380 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55785
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:55:36.011102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:36.097307 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:36.097512 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:36.097535 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:36.098619 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:36.098651 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:36.098676 18790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:36.100072 18790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:36.100268 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:36.100440 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:36.105168 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.105180 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.105167 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.105556 18790 server_base.cc:1034] running on GCE node
I20260928 17:55:36.105784 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:36.106024 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:36.107185 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790618136107176 us; error 32 us; skew 500 ppm
Sep 28 17:55:36 dist-test-slave-zqlt krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:36.109673 18790 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:55:36.110138 18790 webserver.cc:460] Webserver started at http://127.0.98.3:45993/ using document root <none> and password file <none>
I20260928 17:55:36.110381 18790 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:36.110450 18790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:36.110590 18790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:36.111608 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "80657b74162e40f5ac052a5631af79f4"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "b7110d1471ea8ea532fb5c27170ac44d"
server_key_iv: "781c4e75877be4f30a1c79e912841d0b"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.111974 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "80657b74162e40f5ac052a5631af79f4"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "b7110d1471ea8ea532fb5c27170ac44d"
server_key_iv: "781c4e75877be4f30a1c79e912841d0b"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.113272 18790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:36.114013 18805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.114198 18790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:36.114289 18790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "80657b74162e40f5ac052a5631af79f4"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "b7110d1471ea8ea532fb5c27170ac44d"
server_key_iv: "781c4e75877be4f30a1c79e912841d0b"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.114387 18790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:36.130893 18790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:36.132928 18790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:36.133090 18790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:36.133347 18790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:36.133675 18790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:36.133729 18790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.133773 18790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:36.133797 18790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.139737 18790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41349
I20260928 17:55:36.139782 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41349 every 8 connection(s)
I20260928 17:55:36.140179 18790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:55:36 dist-test-slave-zqlt krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:36.147357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18790
I20260928 17:55:36.147495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:36.147759 392 external_mini_cluster.cc:1428] Setting key 9d3b273e5bc0a48f18d1760d3d20ee67
I20260928 17:55:36.150535 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.141003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35461 (local address 127.0.98.62:41497)
0928 17:55:36.141084 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.141087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.141227 (+ 140us) server_negotiation.cc:407] Connection header received
0928 17:55:36.142180 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.142182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.142193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.142239 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.142943 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.143427 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.144553 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.144732 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.144785 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.147009 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.147011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.147011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:36.147018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.148999 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:36.149402 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:36.149403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:36.149403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:36.149462 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:36.149821 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:36.149822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:36.149822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:36.149950 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.150017 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.150399 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.150473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 17:55:36.150758 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.140943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.98.3:35461)
0928 17:55:36.141094 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.141139 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.142081 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.142350 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.142354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.142659 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:36.142861 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.142866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.143524 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.143525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.144478 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.144481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.144819 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.144820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.144921 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.145356 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:36.145364 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:36.146900 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:36.149092 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:36.149097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:36.149099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:36.149316 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:36.149555 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:36.149556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:36.149556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:36.149714 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:36.150024 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:36.150030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:36.150210 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.150353 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.150486 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260928 17:55:36.151260 18919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41497
I20260928 17:55:36.151357 18919 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:36.151508 18919 heartbeater.cc:511] Master 127.0.98.62:41497 requested a full tablet report, sending...
I20260928 17:55:36.151839 18460 ts_manager.cc:194] Registered new tserver with Master: 80657b74162e40f5ac052a5631af79f4 (127.0.98.3:41349)
I20260928 17:55:36.152259 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35461
I20260928 17:55:36.159704 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:36.167062 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.161005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.62:41497)
0928 17:55:36.161086 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.161089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.161161 (+ 72us) server_negotiation.cc:407] Connection header received
0928 17:55:36.161237 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.161238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.161247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.161323 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.161887 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.162359 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.163107 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.163294 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.163399 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.164676 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.164677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.164678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:36.164684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.165878 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:36.166250 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:36.166251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:36.166252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:36.166310 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:36.166533 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:36.166535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:36.166535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:36.166672 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.166786 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.166905 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.166966 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 17:55:36.168613 18460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:36.170097 18460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:36.177006 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.172761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41349 (local address 127.0.0.1:54214)
0928 17:55:36.172899 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.172929 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.172996 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.173642 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.173645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.173647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.173720 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.173722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.174661 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.174662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.175797 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.175800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176726 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.176804 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.176817 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.176888 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.176932 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":39,"thread_start_us":68,"threads_started":1}
I20260928 17:55:36.177131 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.172653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.98.2:40929)
0928 17:55:36.173371 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.173377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.173416 (+ 39us) server_negotiation.cc:407] Connection header received
0928 17:55:36.173460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.173462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.173505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.173848 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.173916 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.174633 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176022 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176733 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176841 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.176898 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.177011 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.177050 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":130,"threads_started":1}
I20260928 17:55:36.177227 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.172555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40929 (local address 127.0.0.1:48868)
0928 17:55:36.172887 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.172928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.172994 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.173682 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.173683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.173686 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.173765 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.173768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.174768 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.174769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.175935 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.175938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176854 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.176933 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.176937 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.176997 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.177038 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":9,"thread_start_us":85,"threads_started":1}
I20260928 17:55:36.177242 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.173261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.3:41349)
0928 17:55:36.173407 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.173412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.173422 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:55:36.173459 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.173461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.173478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.173552 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.173800 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.174565 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.175898 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176628 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.176770 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.177124 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.177169 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 17:55:36.177953 18853 tablet_service.cc:1467] Processing CreateTablet for tablet e363e5a1eabd415481489ef6a5604ced (DEFAULT_TABLE table=test-table [id=5ffd3f03327c4397b386eae696c67edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.178027 18719 tablet_service.cc:1467] Processing CreateTablet for tablet e363e5a1eabd415481489ef6a5604ced (DEFAULT_TABLE table=test-table [id=5ffd3f03327c4397b386eae696c67edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.178351 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e363e5a1eabd415481489ef6a5604ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.178363 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e363e5a1eabd415481489ef6a5604ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.178730 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.172555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45423 (local address 127.0.0.1:35404)
0928 17:55:36.172887 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.172928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.172994 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.174998 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.174999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.175001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.175037 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.175039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176239 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.176240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.177522 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.177526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.178486 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.178487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.178548 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.178551 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.178600 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.178637 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":63,"threads_started":1}
I20260928 17:55:36.179023 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.172665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.0.98.1:45423)
0928 17:55:36.174843 (+ 2178us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.174847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.174855 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:36.174891 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.174893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.174908 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.174987 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.175101 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.176139 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.177597 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.178317 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.178390 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.178448 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.178901 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.178947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2139,"thread_start_us":93,"threads_started":1}
I20260928 17:55:36.179651 18585 tablet_service.cc:1467] Processing CreateTablet for tablet e363e5a1eabd415481489ef6a5604ced (DEFAULT_TABLE table=test-table [id=5ffd3f03327c4397b386eae696c67edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.180025 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e363e5a1eabd415481489ef6a5604ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.181692 18936 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Bootstrap starting.
I20260928 17:55:36.182101 18937 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Bootstrap starting.
I20260928 17:55:36.182476 18936 tablet_bootstrap.cc:654] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.182735 18937 tablet_bootstrap.cc:654] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.183038 18936 log.cc:824] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.183337 18937 log.cc:824] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.183650 18939 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: Bootstrap starting.
I20260928 17:55:36.183898 18936 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: No bootstrap required, opened a new log
I20260928 17:55:36.183966 18936 ts_tablet_manager.cc:1397] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:36.184026 18937 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: No bootstrap required, opened a new log
I20260928 17:55:36.184067 18937 ts_tablet_manager.cc:1397] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:36.184332 18939 tablet_bootstrap.cc:654] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.184867 18939 log.cc:824] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.185648 18939 tablet_bootstrap.cc:492] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: No bootstrap required, opened a new log
I20260928 17:55:36.185715 18939 ts_tablet_manager.cc:1397] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:36.185916 18937 raft_consensus.cc:348] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.186080 18937 raft_consensus.cc:374] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.186123 18937 raft_consensus.cc:729] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c9186bf4fd4eaca1c5c8a8cc669334, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.186136 18936 raft_consensus.cc:348] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.186244 18937 consensus_queue.cc:260] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.186304 18936 raft_consensus.cc:374] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.186357 18936 raft_consensus.cc:729] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80657b74162e40f5ac052a5631af79f4, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.186458 18936 consensus_queue.cc:260] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.186518 18937 ts_tablet_manager.cc:1428] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:36.186654 18785 heartbeater.cc:503] Master 127.0.98.62:41497 was elected leader, sending a full tablet report...
I20260928 17:55:36.186702 18936 ts_tablet_manager.cc:1428] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:36.186978 18919 heartbeater.cc:503] Master 127.0.98.62:41497 was elected leader, sending a full tablet report...
I20260928 17:55:36.187757 18939 raft_consensus.cc:348] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.187937 18939 raft_consensus.cc:374] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.187999 18939 raft_consensus.cc:729] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12010470bcde4ab5a5293d24b9d8a64b, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.188133 18939 consensus_queue.cc:260] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.188395 18939 ts_tablet_manager.cc:1428] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 17:55:36.188432 18651 heartbeater.cc:503] Master 127.0.98.62:41497 was elected leader, sending a full tablet report...
I20260928 17:55:36.209450 18944 raft_consensus.cc:482] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:36.209570 18944 raft_consensus.cc:504] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.210091 18944 leader_election.cc:288] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c9186bf4fd4eaca1c5c8a8cc669334 (127.0.98.2:40929), 80657b74162e40f5ac052a5631af79f4 (127.0.98.3:41349)
I20260928 17:55:36.212955 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.210263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41349 (local address 127.0.98.1:43793)
0928 17:55:36.210323 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.210337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.210382 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.210576 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.210576 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.210578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.210648 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.210650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.211321 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.211322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.212064 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.212066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.212688 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.212772 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.212775 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.212835 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.212879 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:55:36.212954 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.210352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43793 (local address 127.0.98.3:41349)
0928 17:55:36.210423 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.210425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.210436 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:55:36.210476 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.210477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.210493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.210544 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.210766 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.211235 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212214 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.212607 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212690 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.212700 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.212854 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.212893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:55:36.213135 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.210264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40929 (local address 127.0.98.1:46237)
0928 17:55:36.210626 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.210635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.210701 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.210831 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.210832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.210833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.210904 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.210906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.211510 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.211511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.212322 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.212324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.212874 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.212947 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.212950 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.213017 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.213053 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":169,"thread_start_us":126,"threads_started":1}
I20260928 17:55:36.213125 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.210374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46237 (local address 127.0.98.2:40929)
0928 17:55:36.210431 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.210433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.210701 (+ 268us) server_negotiation.cc:407] Connection header received
0928 17:55:36.210759 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.210760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.210771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.210827 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.210977 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.211430 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212409 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.212789 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.212872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.212881 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.213024 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.213065 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:55:36.213348 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e363e5a1eabd415481489ef6a5604ced" candidate_uuid: "12010470bcde4ab5a5293d24b9d8a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80657b74162e40f5ac052a5631af79f4" is_pre_election: true
I20260928 17:55:36.213513 18873 raft_consensus.cc:2454] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12010470bcde4ab5a5293d24b9d8a64b in term 0.
I20260928 17:55:36.213474 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e363e5a1eabd415481489ef6a5604ced" candidate_uuid: "12010470bcde4ab5a5293d24b9d8a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" is_pre_election: true
I20260928 17:55:36.213621 18739 raft_consensus.cc:2454] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12010470bcde4ab5a5293d24b9d8a64b in term 0.
I20260928 17:55:36.213797 18538 leader_election.cc:302] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12010470bcde4ab5a5293d24b9d8a64b, 80657b74162e40f5ac052a5631af79f4; no voters:
I20260928 17:55:36.213925 18944 raft_consensus.cc:2781] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:36.213976 18944 raft_consensus.cc:482] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:36.213994 18944 raft_consensus.cc:3037] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.214700 18944 raft_consensus.cc:504] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.214867 18944 leader_election.cc:288] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [CANDIDATE]: Term 1 election: Requested vote from peers 43c9186bf4fd4eaca1c5c8a8cc669334 (127.0.98.2:40929), 80657b74162e40f5ac052a5631af79f4 (127.0.98.3:41349)
I20260928 17:55:36.215034 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e363e5a1eabd415481489ef6a5604ced" candidate_uuid: "12010470bcde4ab5a5293d24b9d8a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334"
I20260928 17:55:36.215132 18739 raft_consensus.cc:3037] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.215833 18739 raft_consensus.cc:2454] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12010470bcde4ab5a5293d24b9d8a64b in term 1.
I20260928 17:55:36.215973 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e363e5a1eabd415481489ef6a5604ced" candidate_uuid: "12010470bcde4ab5a5293d24b9d8a64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80657b74162e40f5ac052a5631af79f4"
I20260928 17:55:36.216115 18539 leader_election.cc:302] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12010470bcde4ab5a5293d24b9d8a64b, 43c9186bf4fd4eaca1c5c8a8cc669334; no voters:
I20260928 17:55:36.216102 18873 raft_consensus.cc:3037] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.216231 18944 raft_consensus.cc:2781] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:36.216460 18944 raft_consensus.cc:686] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [term 1 LEADER]: Becoming Leader. State: Replica: 12010470bcde4ab5a5293d24b9d8a64b, State: Running, Role: LEADER
I20260928 17:55:36.216580 18944 consensus_queue.cc:237] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } }
I20260928 17:55:36.216969 18873 raft_consensus.cc:2454] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12010470bcde4ab5a5293d24b9d8a64b in term 1.
I20260928 17:55:36.217599 18460 catalog_manager.cc:5495] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 12010470bcde4ab5a5293d24b9d8a64b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "12010470bcde4ab5a5293d24b9d8a64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12010470bcde4ab5a5293d24b9d8a64b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 } health_report { overall_health: UNKNOWN } } }
W20260928 17:55:36.226590 18786 tablet.cc:2369] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:55:36.230154 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.227425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.0.98.1:45423)
0928 17:55:36.227485 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.227487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.227498 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:55:36.227571 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.227572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.227599 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.227738 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:36.227906 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.228467 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.229287 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.229506 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.229591 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.229752 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:36.229859 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:36.229986 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.230028 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.230042 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 17:55:36.232198 18739 raft_consensus.cc:1261] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 1 FOLLOWER]: Refusing update from remote peer 12010470bcde4ab5a5293d24b9d8a64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:36.232191 18873 raft_consensus.cc:1261] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 1 FOLLOWER]: Refusing update from remote peer 12010470bcde4ab5a5293d24b9d8a64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:36.232612 18946 consensus_queue.cc:1035] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c9186bf4fd4eaca1c5c8a8cc669334" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:36.232774 18944 consensus_queue.cc:1035] T e363e5a1eabd415481489ef6a5604ced P 12010470bcde4ab5a5293d24b9d8a64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80657b74162e40f5ac052a5631af79f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:36.235708 18950 mvcc.cc:205] Tried to move back new op lower bound from 7334371886003662848 to 7334371885943959552. Current Snapshot: MvccSnapshot[applied={T|T < 7334371886003662848}]
I20260928 17:55:36.237589 18459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:36.237686 18459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:36.238440 18459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6cc0f6f7374d16ac79c4df47e12bdc: Sending DeleteTablet for 3 replicas of tablet e363e5a1eabd415481489ef6a5604ced
I20260928 17:55:36.238801 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet e363e5a1eabd415481489ef6a5604ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48868
I20260928 17:55:36.238821 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet e363e5a1eabd415481489ef6a5604ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35404
I20260928 17:55:36.238807 18853 tablet_service.cc:1514] Processing DeleteTablet for tablet e363e5a1eabd415481489ef6a5604ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54214
I20260928 17:55:36.239059 18961 tablet_replica.cc:330] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: stopping tablet replica
I20260928 17:55:36.239156 18961 raft_consensus.cc:2229] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:36.239249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18522
I20260928 17:55:36.239264 18961 raft_consensus.cc:2258] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:36.240943 18961 ts_tablet_manager.cc:1905] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:36.241742 18959 tablet_replica.cc:330] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: stopping tablet replica
I20260928 17:55:36.241848 18959 raft_consensus.cc:2229] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:36.241956 18959 raft_consensus.cc:2258] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:36.242347 18959 ts_tablet_manager.cc:1905] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:36.242884 18961 ts_tablet_manager.cc:1918] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:36.242959 18961 log.cc:1196] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/e363e5a1eabd415481489ef6a5604ced
I20260928 17:55:36.243234 18961 ts_tablet_manager.cc:1939] T e363e5a1eabd415481489ef6a5604ced P 80657b74162e40f5ac052a5631af79f4: Deleting consensus metadata
I20260928 17:55:36.243721 18959 ts_tablet_manager.cc:1918] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:36.243804 18959 log.cc:1196] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/e363e5a1eabd415481489ef6a5604ced
I20260928 17:55:36.243901 18444 catalog_manager.cc:4841] TS 80657b74162e40f5ac052a5631af79f4 (127.0.98.3:41349): tablet e363e5a1eabd415481489ef6a5604ced (table test-table [id=5ffd3f03327c4397b386eae696c67edd]) successfully deleted
I20260928 17:55:36.244136 18959 ts_tablet_manager.cc:1939] T e363e5a1eabd415481489ef6a5604ced P 43c9186bf4fd4eaca1c5c8a8cc669334: Deleting consensus metadata
I20260928 17:55:36.244525 18445 catalog_manager.cc:4841] TS 43c9186bf4fd4eaca1c5c8a8cc669334 (127.0.98.2:40929): tablet e363e5a1eabd415481489ef6a5604ced (table test-table [id=5ffd3f03327c4397b386eae696c67edd]) successfully deleted
W20260928 17:55:36.246074 18444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45423 (error 108)
I20260928 17:55:36.246304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18656
I20260928 17:55:36.246963 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.246451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45423 (local address 127.0.0.1:35414)
0928 17:55:36.246892 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.246934 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
I20260928 17:55:36.251197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18790
W20260928 17:55:36.251183 18444 catalog_manager.cc:4584] TS 12010470bcde4ab5a5293d24b9d8a64b (127.0.98.1:45423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e363e5a1eabd415481489ef6a5604ced: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45423: connect: Connection refused (error 111)
I20260928 17:55:36.255788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18428
2026-09-28T17:55:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T17:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:36Z Disabled control of system clock
I20260928 17:55:36.283646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46015
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:36.363031 18966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:36.363283 18966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:36.363306 18966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:36.364603 18966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:36.364647 18966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:36.364660 18966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:36.364696 18966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:36.364725 18966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:36.366102 18966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:36.366376 18966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:36.366693 18966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:36.371510 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.371579 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.371533 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.371822 18966 server_base.cc:1034] running on GCE node
I20260928 17:55:36.372135 18966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:36.372481 18966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:36.373647 18966 hybrid_clock.cc:648] HybridClock initialized: now 1790618136373637 us; error 28 us; skew 500 ppm
I20260928 17:55:36.374936 18966 webserver.cc:460] Webserver started at http://127.0.98.62:42191/ using document root <none> and password file <none>
I20260928 17:55:36.375259 18966 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:36.375331 18966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:36.375478 18966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:36.376734 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "8ee21fb29ee443f0b1e9311bfe606cf9"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "30641086b6b8c9186b5a646dd8b3a840"
server_key_iv: "40d42cdda1cb9592fb832b2ebead6e5e"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.377123 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ee21fb29ee443f0b1e9311bfe606cf9"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "30641086b6b8c9186b5a646dd8b3a840"
server_key_iv: "40d42cdda1cb9592fb832b2ebead6e5e"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.378433 18966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:36.379170 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.379349 18966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:36.379447 18966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "8ee21fb29ee443f0b1e9311bfe606cf9"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "30641086b6b8c9186b5a646dd8b3a840"
server_key_iv: "40d42cdda1cb9592fb832b2ebead6e5e"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.379534 18966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:36.398412 18966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:36.398753 18966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:36.398919 18966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:36.403005 18966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46015
I20260928 17:55:36.403043 19032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46015 every 8 connection(s)
I20260928 17:55:36.403390 18966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:36.404369 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.406586 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: Bootstrap starting.
I20260928 17:55:36.407174 19033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.407424 19033 log.cc:824] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.407836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 18966
I20260928 17:55:36.407946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:36.408154 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: No bootstrap required, opened a new log
I20260928 17:55:36.408181 392 external_mini_cluster.cc:1428] Setting key 1a4e3aac9c92e33241704e47f299826a
I20260928 17:55:36.410742 19033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260928 17:55:36.410950 19033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.410988 19033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee21fb29ee443f0b1e9311bfe606cf9, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.411121 19033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260928 17:55:36.411201 19033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:36.411233 19033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:36.411278 19033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.412007 19033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260928 17:55:36.412158 19033 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee21fb29ee443f0b1e9311bfe606cf9; no voters:
I20260928 17:55:36.412374 19033 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:36.412405 19038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:36.412511 19038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee21fb29ee443f0b1e9311bfe606cf9, State: Running, Role: LEADER
I20260928 17:55:36.412596 19038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260928 17:55:36.412662 19033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:36.413359 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee21fb29ee443f0b1e9311bfe606cf9. Latest consensus state: current_term: 1 leader_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } } }
I20260928 17:55:36.413455 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:36.413614 19039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee21fb29ee443f0b1e9311bfe606cf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } } }
I20260928 17:55:36.413664 19039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:36.414055 19047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:36.414407 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.408704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.0.98.62:46015)
0928 17:55:36.408969 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.408976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.409001 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:55:36.409436 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.409443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.409501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.409614 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.409879 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.410753 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.411838 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.412387 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.412905 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.413385 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.413389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.413390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:36.413599 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.413641 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:36.413656 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.413719 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.414092 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.414187 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":128,"threads_started":1}
I20260928 17:55:36.414578 19047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:36.416186 19047 catalog_manager.cc:1347] Generated new cluster ID: c577cef25dbb41dfaa9553e724235c46
I20260928 17:55:36.416239 19047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:36.426088 19047 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:36.426819 19047 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:36.435052 19047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: Generated new TSK 0
I20260928 17:55:36.435310 19047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:36.442202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--builtin_ntp_servers=127.0.98.20:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:36.524019 19057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:36.524302 19057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:36.524339 19057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:36.525568 19057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:36.525636 19057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:36.525682 19057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:36.527148 19057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:36.527421 19057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:36.527729 19057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:36.533663 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.533737 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.533663 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.533957 19057 server_base.cc:1034] running on GCE node
I20260928 17:55:36.534181 19057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:36.534459 19057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:36.535641 19057 hybrid_clock.cc:648] HybridClock initialized: now 1790618136535620 us; error 35 us; skew 500 ppm
I20260928 17:55:36.537118 19057 webserver.cc:460] Webserver started at http://127.0.98.1:45039/ using document root <none> and password file <none>
I20260928 17:55:36.537380 19057 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:36.537457 19057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:36.537617 19057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:36.538899 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "0377018f5a3d4349ae7f1a18b2e8d364"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "84cf5a552f75cd2ea99f98e83f0d261a"
server_key_iv: "d3a9798ab3129fb2abbd27d5c3fde566"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.539321 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0377018f5a3d4349ae7f1a18b2e8d364"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "84cf5a552f75cd2ea99f98e83f0d261a"
server_key_iv: "d3a9798ab3129fb2abbd27d5c3fde566"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.540797 19057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 17:55:36.541603 19071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.541823 19057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:36.541939 19057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "0377018f5a3d4349ae7f1a18b2e8d364"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "84cf5a552f75cd2ea99f98e83f0d261a"
server_key_iv: "d3a9798ab3129fb2abbd27d5c3fde566"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.542019 19057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:36.550766 19057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:36.551033 19057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:36.551174 19057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:36.551386 19057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:36.551684 19057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:36.551741 19057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.551781 19057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:36.551810 19057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.557466 19057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43527
I20260928 17:55:36.557547 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43527 every 8 connection(s)
I20260928 17:55:36.557852 19057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:55:36.562906 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.558601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49369 (local address 127.0.98.62:46015)
0928 17:55:36.558681 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.558683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.558875 (+ 192us) server_negotiation.cc:407] Connection header received
0928 17:55:36.559344 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.559345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.559358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.559409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.560051 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.560605 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.561611 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.561816 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.561881 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.562670 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.562671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.562671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:36.562675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.562687 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:36.562691 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.562741 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.562823 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.562847 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:55:36.563087 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.558557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.1:49369)
0928 17:55:36.558733 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.558793 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.559264 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.559471 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.559474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.559482 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:36.559979 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.559983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.560699 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.560700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.561519 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.561521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.561904 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.561904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.562015 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.562547 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:36.562547 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:36.562574 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:36.562596 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:36.562753 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:36.562757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:36.562760 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.562811 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.562892 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 17:55:36.563589 19185 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260928 17:55:36.563685 19185 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:36.563859 19185 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260928 17:55:36.564404 18997 ts_manager.cc:194] Registered new tserver with Master: 0377018f5a3d4349ae7f1a18b2e8d364 (127.0.98.1:43527)
I20260928 17:55:36.565023 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49369
I20260928 17:55:36.567662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19057
I20260928 17:55:36.567796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:36.568022 392 external_mini_cluster.cc:1428] Setting key aee5707f055fe70483b5b2c215270c30
I20260928 17:55:36.568864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--builtin_ntp_servers=127.0.98.20:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:36.651110 19188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:36.651355 19188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:36.651382 19188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:36.652594 19188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:36.652633 19188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:36.652659 19188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:36.653982 19188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:36.654172 19188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:36.654434 19188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:36.659417 19196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.659540 19188 server_base.cc:1034] running on GCE node
W20260928 17:55:36.659441 19193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.659442 19194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.659847 19188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:36.660179 19188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:36.661321 19188 hybrid_clock.cc:648] HybridClock initialized: now 1790618136661301 us; error 37 us; skew 500 ppm
I20260928 17:55:36.662528 19188 webserver.cc:460] Webserver started at http://127.0.98.2:36253/ using document root <none> and password file <none>
I20260928 17:55:36.662796 19188 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:36.662843 19188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:36.662973 19188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:36.664042 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "facd93f0624a4daa812f3abef6abfdfd"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "94ad1f2b46f376d2520be735959c3d7b"
server_key_iv: "81abe77a6c79617bfefa7816debe88d6"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.664444 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "facd93f0624a4daa812f3abef6abfdfd"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "94ad1f2b46f376d2520be735959c3d7b"
server_key_iv: "81abe77a6c79617bfefa7816debe88d6"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.665747 19188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:36.666447 19202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.666608 19188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:36.666723 19188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "facd93f0624a4daa812f3abef6abfdfd"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "94ad1f2b46f376d2520be735959c3d7b"
server_key_iv: "81abe77a6c79617bfefa7816debe88d6"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.666800 19188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:36.683391 19188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:36.683678 19188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:36.683820 19188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:36.684041 19188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:36.684414 19188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:36.684473 19188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.684514 19188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:36.684540 19188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.690181 19188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37257
I20260928 17:55:36.690223 19315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37257 every 8 connection(s)
I20260928 17:55:36.690548 19188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:55:36.693681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19188
I20260928 17:55:36.693812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:36.694001 392 external_mini_cluster.cc:1428] Setting key be8735016cd95cf87821cd1fbfb61751
I20260928 17:55:36.695043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--builtin_ntp_servers=127.0.98.20:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 17:55:36.696384 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.691296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51799 (local address 127.0.98.62:46015)
0928 17:55:36.691346 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.691348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.691584 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:55:36.692172 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.692174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.692186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.692259 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.692943 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.693451 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.694722 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.695022 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.695102 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.696053 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.696054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.696054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:36.696060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.696076 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:36.696081 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.696175 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.696240 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.696270 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:55:36.696573 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.691260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.2:51799)
0928 17:55:36.691463 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.691496 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.692030 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.692332 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.692336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.692346 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:36.692844 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.692848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.693557 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.693558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.694629 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.694632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.695104 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.695105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.695257 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.695910 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:36.695911 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:36.695942 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:36.695965 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:36.696178 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:36.696182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:36.696184 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.696238 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.696313 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260928 17:55:36.697080 19316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260928 17:55:36.697185 19316 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:36.697365 19316 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260928 17:55:36.697795 18997 ts_manager.cc:194] Registered new tserver with Master: facd93f0624a4daa812f3abef6abfdfd (127.0.98.2:37257)
I20260928 17:55:36.698164 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51799
W20260928 17:55:36.776189 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:36.776414 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:36.776437 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:36.777637 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:36.777685 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:36.777710 19319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:36.779090 19319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:36.779350 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:36.779681 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:36.785143 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:36.785133 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.785162 19319 server_base.cc:1034] running on GCE node
W20260928 17:55:36.785115 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:36.785526 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:36.785763 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:36.786917 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790618136786901 us; error 33 us; skew 500 ppm
I20260928 17:55:36.788175 19319 webserver.cc:460] Webserver started at http://127.0.98.3:39899/ using document root <none> and password file <none>
I20260928 17:55:36.788444 19319 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:36.788489 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:36.788640 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:36.789767 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6f5c9a7901f43debfa22b203aab3719"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "d087e1e66de24fbc610e9c9c9d3aceac"
server_key_iv: "40d65980fa4f45126d8ce372eaf5a20d"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.790131 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6f5c9a7901f43debfa22b203aab3719"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "d087e1e66de24fbc610e9c9c9d3aceac"
server_key_iv: "40d65980fa4f45126d8ce372eaf5a20d"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.791484 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:36.792284 19333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.792519 19319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:36.792608 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "a6f5c9a7901f43debfa22b203aab3719"
format_stamp: "Formatted at 2026-09-28 17:55:36 on dist-test-slave-zqlt"
server_key: "d087e1e66de24fbc610e9c9c9d3aceac"
server_key_iv: "40d65980fa4f45126d8ce372eaf5a20d"
server_key_version: "encryptionkey@0"
I20260928 17:55:36.792681 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:36.810832 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:36.811189 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:36.811339 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:36.811607 19319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:36.811972 19319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:36.812047 19319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.812105 19319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:36.812120 19319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:36.818091 19319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33543
I20260928 17:55:36.818157 19446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33543 every 8 connection(s)
I20260928 17:55:36.818498 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:55:36.820568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19319
I20260928 17:55:36.820650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:36.820902 392 external_mini_cluster.cc:1428] Setting key faadcbcc47c865964b24b6b6b710e486
I20260928 17:55:36.824213 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.819286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56141 (local address 127.0.98.62:46015)
0928 17:55:36.819336 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.819338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.819600 (+ 262us) server_negotiation.cc:407] Connection header received
0928 17:55:36.820191 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.820192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.820204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.820261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.821078 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.821797 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.822800 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.822996 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.823051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.823915 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.823916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.823917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:36.823921 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.823935 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:36.823938 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.823983 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.824130 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.824153 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:55:36.824496 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.819278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.3:56141)
0928 17:55:36.819471 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.819506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.820037 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.820346 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.820349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.820358 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:36.820980 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.820989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.821898 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.821899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.822727 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.822730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.823093 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.823094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.823236 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.823778 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:36.823779 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:36.823809 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:36.823833 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:36.824017 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:36.824023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:36.824026 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.824145 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.824247 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 17:55:36.824985 19447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260928 17:55:36.825106 19447 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:36.825299 19447 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260928 17:55:36.825697 18997 ts_manager.cc:194] Registered new tserver with Master: a6f5c9a7901f43debfa22b203aab3719 (127.0.98.3:33543)
I20260928 17:55:36.826063 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56141
I20260928 17:55:36.832876 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:36.837414 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.834324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.98.62:46015)
0928 17:55:36.834391 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.834393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.834570 (+ 177us) server_negotiation.cc:407] Connection header received
0928 17:55:36.834607 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.834607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.834616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.834665 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:36.834881 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.835367 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.836153 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.836326 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.836375 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.837117 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:36.837118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:36.837118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:36.837123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:36.837136 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:36.837139 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:36.837214 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.837283 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.837293 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 17:55:36.839092 18997 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:36.840694 18997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:36.847788 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.0.98.2:37257)
0928 17:55:36.843651 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.843656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.844199 (+ 543us) server_negotiation.cc:407] Connection header received
0928 17:55:36.844245 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.844247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.844262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.844346 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.844535 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845634 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.846859 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847366 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.847458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.847557 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.847637 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.847677 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260928 17:55:36.847826 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.0.1:54920)
0928 17:55:36.843588 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.843619 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.843714 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.843965 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.843968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.843971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.844111 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.844114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845527 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.845528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.846543 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.846546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847534 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.847536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.847601 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.847605 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.847640 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.847682 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":92,"threads_started":1}
I20260928 17:55:36.848029 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.0.98.3:33543)
0928 17:55:36.843504 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.843508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.843940 (+ 432us) server_negotiation.cc:407] Connection header received
0928 17:55:36.844223 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.844226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.844243 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.844327 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.844453 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845306 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.846820 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847528 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.847596 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.847686 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.847882 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.847931 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260928 17:55:36.847836 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37257 (local address 127.0.0.1:36862)
0928 17:55:36.843988 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.843998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.844051 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.844409 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.844410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.844412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.844453 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.844454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845713 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.845714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.846778 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.846780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847459 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.847460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.847532 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.847545 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.847622 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.847677 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":67,"threads_started":1}
I20260928 17:55:36.848348 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.0.98.1:43527)
0928 17:55:36.843430 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.843436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.843735 (+ 299us) server_negotiation.cc:407] Connection header received
0928 17:55:36.843849 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.843851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.843872 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.843962 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.844179 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845398 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.846637 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847407 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.848055 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.848189 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.848218 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.848265 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":145,"threads_started":1}
I20260928 17:55:36.848376 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.843084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.0.1:38822)
0928 17:55:36.843877 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.843886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.844136 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.844340 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.844341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.844343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.844384 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.844386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.845395 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.845396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.846714 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.846717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.847722 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.847723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.847780 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.847783 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.848260 (+ 477us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.848307 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":32,"thread_start_us":64,"threads_started":1}
I20260928 17:55:36.849088 19250 tablet_service.cc:1467] Processing CreateTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 (DEFAULT_TABLE table=test-table [id=f793ea0a93e740fd94a792cadde5f309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.849112 19119 tablet_service.cc:1467] Processing CreateTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 (DEFAULT_TABLE table=test-table [id=f793ea0a93e740fd94a792cadde5f309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.849471 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dbbb1dbee0409cb810c055694f61c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.849471 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dbbb1dbee0409cb810c055694f61c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.849370 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 (DEFAULT_TABLE table=test-table [id=f793ea0a93e740fd94a792cadde5f309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:36.849706 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19dbbb1dbee0409cb810c055694f61c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:36.852397 19461 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Bootstrap starting.
I20260928 17:55:36.853076 19462 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Bootstrap starting.
I20260928 17:55:36.853166 19461 tablet_bootstrap.cc:654] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.853503 19461 log.cc:824] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.853744 19462 tablet_bootstrap.cc:654] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.854087 19462 log.cc:824] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.854287 19461 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: No bootstrap required, opened a new log
I20260928 17:55:36.854370 19461 ts_tablet_manager.cc:1397] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:55:36.854362 19463 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: Bootstrap starting.
I20260928 17:55:36.854919 19462 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: No bootstrap required, opened a new log
I20260928 17:55:36.855000 19463 tablet_bootstrap.cc:654] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:36.855106 19462 ts_tablet_manager.cc:1397] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:36.855342 19463 log.cc:824] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:36.856278 19463 tablet_bootstrap.cc:492] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: No bootstrap required, opened a new log
I20260928 17:55:36.856350 19463 ts_tablet_manager.cc:1397] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 17:55:36.857407 19462 raft_consensus.cc:348] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.857429 19461 raft_consensus.cc:348] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.857576 19462 raft_consensus.cc:374] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.857617 19461 raft_consensus.cc:374] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.857621 19462 raft_consensus.cc:729] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facd93f0624a4daa812f3abef6abfdfd, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.857656 19461 raft_consensus.cc:729] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f5c9a7901f43debfa22b203aab3719, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.857769 19462 consensus_queue.cc:260] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.857790 19461 consensus_queue.cc:260] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.858088 19462 ts_tablet_manager.cc:1428] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:36.858127 19316 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
I20260928 17:55:36.858063 19461 ts_tablet_manager.cc:1428] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 17:55:36.858493 19447 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
I20260928 17:55:36.858922 19463 raft_consensus.cc:348] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.859090 19463 raft_consensus.cc:374] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:36.859134 19463 raft_consensus.cc:729] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0377018f5a3d4349ae7f1a18b2e8d364, State: Initialized, Role: FOLLOWER
I20260928 17:55:36.859246 19463 consensus_queue.cc:260] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.859525 19463 ts_tablet_manager.cc:1428] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:36.859572 19185 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
I20260928 17:55:36.939363 19468 raft_consensus.cc:482] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:36.939491 19468 raft_consensus.cc:504] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.939915 19468 leader_election.cc:288] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0377018f5a3d4349ae7f1a18b2e8d364 (127.0.98.1:43527), a6f5c9a7901f43debfa22b203aab3719 (127.0.98.3:33543)
W20260928 17:55:36.941038 19317 tablet.cc:2369] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 17:55:36.942899 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.940148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58125 (local address 127.0.98.1:43527)
0928 17:55:36.940211 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.940213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.940223 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:55:36.940261 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.940262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.940277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.940341 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.940490 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.941194 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942164 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.942544 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942629 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.942639 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.942786 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.942824 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:55:36.942909 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.940227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43867 (local address 127.0.98.3:33543)
0928 17:55:36.940284 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.940285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.940480 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:55:36.940505 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.940506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.940513 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.940552 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:36.940727 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.941224 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942200 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.942545 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942627 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.942636 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.942806 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.942840 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 17:55:36.942921 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.940052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.98.2:58125)
0928 17:55:36.940116 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.940128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.940174 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.940352 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.940353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.940355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.940417 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.940419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.941288 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.941289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.942070 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.942072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.942629 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.942708 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.942711 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.942781 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.942837 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":17}
I20260928 17:55:36.942932 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.940119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.2:43867)
0928 17:55:36.940275 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.940283 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:36.940334 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:36.940581 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:36.940581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:36.940583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:36.940651 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.940653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.941311 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.941312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.942080 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:36.942082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.942631 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.942632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:36.942707 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.942711 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:36.942795 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 17:55:36.942833 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260928 17:55:36.943287 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dbbb1dbee0409cb810c055694f61c8" candidate_uuid: "facd93f0624a4daa812f3abef6abfdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f5c9a7901f43debfa22b203aab3719" is_pre_election: true
I20260928 17:55:36.943279 19139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dbbb1dbee0409cb810c055694f61c8" candidate_uuid: "facd93f0624a4daa812f3abef6abfdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" is_pre_election: true
I20260928 17:55:36.943454 19401 raft_consensus.cc:2454] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facd93f0624a4daa812f3abef6abfdfd in term 0.
I20260928 17:55:36.943465 19139 raft_consensus.cc:2454] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facd93f0624a4daa812f3abef6abfdfd in term 0.
I20260928 17:55:36.943689 19206 leader_election.cc:302] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0377018f5a3d4349ae7f1a18b2e8d364, facd93f0624a4daa812f3abef6abfdfd; no voters:
I20260928 17:55:36.943831 19468 raft_consensus.cc:2781] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:36.943890 19468 raft_consensus.cc:482] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:36.943908 19468 raft_consensus.cc:3037] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.944698 19468 raft_consensus.cc:504] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.944835 19468 leader_election.cc:288] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [CANDIDATE]: Term 1 election: Requested vote from peers 0377018f5a3d4349ae7f1a18b2e8d364 (127.0.98.1:43527), a6f5c9a7901f43debfa22b203aab3719 (127.0.98.3:33543)
I20260928 17:55:36.944972 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dbbb1dbee0409cb810c055694f61c8" candidate_uuid: "facd93f0624a4daa812f3abef6abfdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f5c9a7901f43debfa22b203aab3719"
I20260928 17:55:36.944993 19139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19dbbb1dbee0409cb810c055694f61c8" candidate_uuid: "facd93f0624a4daa812f3abef6abfdfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0377018f5a3d4349ae7f1a18b2e8d364"
I20260928 17:55:36.945052 19401 raft_consensus.cc:3037] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.945058 19139 raft_consensus.cc:3037] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:36.945849 19401 raft_consensus.cc:2454] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate facd93f0624a4daa812f3abef6abfdfd in term 1.
I20260928 17:55:36.945870 19139 raft_consensus.cc:2454] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate facd93f0624a4daa812f3abef6abfdfd in term 1.
I20260928 17:55:36.946022 19203 leader_election.cc:302] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6f5c9a7901f43debfa22b203aab3719, facd93f0624a4daa812f3abef6abfdfd; no voters:
I20260928 17:55:36.946103 19468 raft_consensus.cc:2781] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:36.946250 19468 raft_consensus.cc:686] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 1 LEADER]: Becoming Leader. State: Replica: facd93f0624a4daa812f3abef6abfdfd, State: Running, Role: LEADER
I20260928 17:55:36.946360 19468 consensus_queue.cc:237] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } }
I20260928 17:55:36.947170 18997 catalog_manager.cc:5495] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd reported cstate change: term changed from 0 to 1, leader changed from <none> to facd93f0624a4daa812f3abef6abfdfd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "facd93f0624a4daa812f3abef6abfdfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "facd93f0624a4daa812f3abef6abfdfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37257 } health_report { overall_health: HEALTHY } } }
I20260928 17:55:36.963053 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.960660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.0.98.2:37257)
0928 17:55:36.960719 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:36.960721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:36.960729 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:36.960814 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:36.960815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:36.960831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:36.960889 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:36.961029 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.961475 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.962308 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:36.962487 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:36.962534 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:36.962685 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:36.962769 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:36.962838 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:36.962919 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 17:55:36.962927 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:55:36.964834 19139 raft_consensus.cc:1261] T 19dbbb1dbee0409cb810c055694f61c8 P 0377018f5a3d4349ae7f1a18b2e8d364 [term 1 FOLLOWER]: Refusing update from remote peer facd93f0624a4daa812f3abef6abfdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:36.964851 19401 raft_consensus.cc:1261] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 1 FOLLOWER]: Refusing update from remote peer facd93f0624a4daa812f3abef6abfdfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:36.965296 19468 consensus_queue.cc:1035] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f5c9a7901f43debfa22b203aab3719" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:36.965461 19471 consensus_queue.cc:1035] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0377018f5a3d4349ae7f1a18b2e8d364" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:36.967734 19474 mvcc.cc:205] Tried to move back new op lower bound from 7334371889005494272 to 7334371888932777984. Current Snapshot: MvccSnapshot[applied={T|T < 7334371889005494272}]
I20260928 17:55:36.968623 19476 mvcc.cc:205] Tried to move back new op lower bound from 7334371889005494272 to 7334371888932777984. Current Snapshot: MvccSnapshot[applied={T|T < 7334371889005494272}]
I20260928 17:55:36.970721 18996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:36.970847 18996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:36.971616 18996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee21fb29ee443f0b1e9311bfe606cf9: Sending DeleteTablet for 3 replicas of tablet 19dbbb1dbee0409cb810c055694f61c8
I20260928 17:55:36.971956 19250 tablet_service.cc:1514] Processing DeleteTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='slave'} at 127.0.0.1:36862
I20260928 17:55:36.971992 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='slave'} at 127.0.0.1:38822
I20260928 17:55:36.971993 19119 tablet_service.cc:1514] Processing DeleteTablet for tablet 19dbbb1dbee0409cb810c055694f61c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:36 UTC) from {username='slave'} at 127.0.0.1:54920
I20260928 17:55:36.972321 19486 tablet_replica.cc:330] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: stopping tablet replica
I20260928 17:55:36.972452 19486 raft_consensus.cc:2229] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:36.972568 19486 raft_consensus.cc:2258] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:36.972637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19057
I20260928 17:55:36.972470 19485 tablet_replica.cc:330] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: stopping tablet replica
I20260928 17:55:36.973609 19485 raft_consensus.cc:2229] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:55:36.973742 19486 ts_tablet_manager.cc:1905] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:36.973845 19485 raft_consensus.cc:2258] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:36.974330 19485 ts_tablet_manager.cc:1905] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:36.975798 19486 ts_tablet_manager.cc:1918] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:36.975883 19486 log.cc:1196] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/19dbbb1dbee0409cb810c055694f61c8
I20260928 17:55:36.976154 19485 ts_tablet_manager.cc:1918] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:36.976173 19486 ts_tablet_manager.cc:1939] T 19dbbb1dbee0409cb810c055694f61c8 P a6f5c9a7901f43debfa22b203aab3719: Deleting consensus metadata
I20260928 17:55:36.976341 19485 log.cc:1196] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/19dbbb1dbee0409cb810c055694f61c8
I20260928 17:55:36.976562 18981 catalog_manager.cc:4841] TS a6f5c9a7901f43debfa22b203aab3719 (127.0.98.3:33543): tablet 19dbbb1dbee0409cb810c055694f61c8 (table test-table [id=f793ea0a93e740fd94a792cadde5f309]) successfully deleted
I20260928 17:55:36.976672 19485 ts_tablet_manager.cc:1939] T 19dbbb1dbee0409cb810c055694f61c8 P facd93f0624a4daa812f3abef6abfdfd: Deleting consensus metadata
I20260928 17:55:36.977053 18981 catalog_manager.cc:4841] TS facd93f0624a4daa812f3abef6abfdfd (127.0.98.2:37257): tablet 19dbbb1dbee0409cb810c055694f61c8 (table test-table [id=f793ea0a93e740fd94a792cadde5f309]) successfully deleted
W20260928 17:55:36.979682 18984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43527 (error 108)
I20260928 17:55:36.980189 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:36.980033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43527 (local address 127.0.0.1:54924)
0928 17:55:36.980114 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:36.980155 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:55:36.980240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19188
W20260928 17:55:36.981614 18984 catalog_manager.cc:4584] TS 0377018f5a3d4349ae7f1a18b2e8d364 (127.0.98.1:43527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19dbbb1dbee0409cb810c055694f61c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43527: connect: Connection refused (error 111)
I20260928 17:55:36.985492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19319
I20260928 17:55:36.990207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 18966
2026-09-28T17:55:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (733 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:37 dist-test-slave-zqlt krb5kdc[19491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:37 dist-test-slave-zqlt krb5kdc[19491](info): set up 2 sockets
Sep 28 17:55:37 dist-test-slave-zqlt krb5kdc[19491](info): commencing operation
krb5kdc: starting...
W20260928 17:55:38.020247 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:38.142585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:38.222041 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:38.222325 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:38.222362 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:38.223582 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:38.223655 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:38.223680 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:38.223701 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:38.223732 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:38.225328 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:38.225646 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:38.225890 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:38.230800 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.231083 19504 server_base.cc:1034] running on GCE node
W20260928 17:55:38.230792 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:38.230805 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.231474 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:38.231762 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:38.232954 19504 hybrid_clock.cc:648] HybridClock initialized: now 1790618138232950 us; error 52 us; skew 500 ppm
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:38.235536 19504 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:38.236017 19504 webserver.cc:460] Webserver started at http://127.0.98.62:37061/ using document root <none> and password file <none>
I20260928 17:55:38.236297 19504 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:38.236370 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:38.236527 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:38.237690 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "da855a94e8112ec1c119c742278774e9"
server_key_iv: "96f2e2e4544f85f39d0bbb104743a94a"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.238076 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "da855a94e8112ec1c119c742278774e9"
server_key_iv: "96f2e2e4544f85f39d0bbb104743a94a"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.239464 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:55:38.240218 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.240378 19504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:38.240481 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "da855a94e8112ec1c119c742278774e9"
server_key_iv: "96f2e2e4544f85f39d0bbb104743a94a"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.240567 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:38.279795 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:38.281930 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:38.282115 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:38.286039 19504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33357
I20260928 17:55:38.286100 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33357 every 8 connection(s)
I20260928 17:55:38.286473 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:38.287400 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:38.287642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19504
I20260928 17:55:38.287776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:38.287993 392 external_mini_cluster.cc:1428] Setting key f0af70bec23b04ebeb33ed680dad5ec3
I20260928 17:55:38.290439 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: Bootstrap starting.
I20260928 17:55:38.291025 19572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:38.291308 19572 log.cc:824] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:38.291985 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: No bootstrap required, opened a new log
I20260928 17:55:38.294042 19572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } }
I20260928 17:55:38.294215 19572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:38.294252 19572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c4bfb5e7c4ecc8abeef216f16bdb1, State: Initialized, Role: FOLLOWER
I20260928 17:55:38.294405 19572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } }
I20260928 17:55:38.294486 19572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:38.294516 19572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:38.294561 19572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:38.295058 19572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } }
I20260928 17:55:38.295140 19572 leader_election.cc:302] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950c4bfb5e7c4ecc8abeef216f16bdb1; no voters:
I20260928 17:55:38.295352 19572 leader_election.cc:288] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:38.295392 19577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:38.295504 19577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [term 1 LEADER]: Becoming Leader. State: Replica: 950c4bfb5e7c4ecc8abeef216f16bdb1, State: Running, Role: LEADER
I20260928 17:55:38.295580 19577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } }
I20260928 17:55:38.295624 19572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:38.296065 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } } }
I20260928 17:55:38.296069 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950c4bfb5e7c4ecc8abeef216f16bdb1. Latest consensus state: current_term: 1 leader_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950c4bfb5e7c4ecc8abeef216f16bdb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33357 } } }
I20260928 17:55:38.296240 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:38.296231 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:38.296525 19586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:38.297073 19586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:38.298437 19586 catalog_manager.cc:1347] Generated new cluster ID: eadc91fb575e49deb30d22efda36a89b
I20260928 17:55:38.298485 19586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:38.299533 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.288557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.0.98.62:33357)
0928 17:55:38.288722 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.288731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.288765 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:55:38.289171 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.289178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.289231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.289353 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:38.290138 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.291113 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.291947 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.292465 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.292593 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.295046 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:38.295050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:38.295051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:38.295063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:38.297928 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.298337 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.298340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.298341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.298454 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.298676 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.298677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.298677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.298913 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:38.299080 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.299179 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.299313 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260928 17:55:38.325003 19586 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:38.325551 19586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:38.328868 19586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: Generated new TSK 0
I20260928 17:55:38.329064 19586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 17:55:38.358789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:38.453840 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:38.454061 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:38.454100 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:38.455247 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:38.455319 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:38.455365 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:38.456938 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:38.457197 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:38.457432 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:38.462291 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.462406 19598 server_base.cc:1034] running on GCE node
W20260928 17:55:38.462306 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:38.462309 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.462733 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:38.462994 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:38.464176 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790618138464164 us; error 26 us; skew 500 ppm
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:38.466852 19598 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 17:55:38.467329 19598 webserver.cc:460] Webserver started at http://127.0.98.1:42101/ using document root <none> and password file <none>
I20260928 17:55:38.467546 19598 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:38.467623 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:38.467782 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:38.468911 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "fdb5f7651c15434db726b68968661908"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "4017f34cbac6ddb22d2a4724911ad16c"
server_key_iv: "649a70c8c8095342789b21d39e446721"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.469264 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fdb5f7651c15434db726b68968661908"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "4017f34cbac6ddb22d2a4724911ad16c"
server_key_iv: "649a70c8c8095342789b21d39e446721"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.471165 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 17:55:38.472124 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.472328 19598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:38.472429 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "fdb5f7651c15434db726b68968661908"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "4017f34cbac6ddb22d2a4724911ad16c"
server_key_iv: "649a70c8c8095342789b21d39e446721"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.472522 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:38.506301 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:38.509301 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:38.509474 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:38.509776 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:38.510176 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:38.510226 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.510284 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:38.510317 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.518059 19598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36123
I20260928 17:55:38.518170 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36123 every 8 connection(s)
I20260928 17:55:38.518535 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:55:38.524806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19598
I20260928 17:55:38.524938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:38.525146 392 external_mini_cluster.cc:1428] Setting key 6a3dd96690ecf79807006d0ebb30fb46
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:38.528808 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.519348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57137 (local address 127.0.98.62:33357)
0928 17:55:38.519397 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.519399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.519587 (+ 188us) server_negotiation.cc:407] Connection header received
0928 17:55:38.520704 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.520706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.520720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.520766 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:38.521473 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.521983 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.523145 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.523396 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.523483 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.525836 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:38.525838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:38.525838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:38.525846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:38.527287 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.527732 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.527734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.527734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.527794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.528058 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.528059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.528059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.528223 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:38.528293 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.528570 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.528667 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:55:38.528843 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.519303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33357 (local address 127.0.98.1:57137)
0928 17:55:38.519450 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.519499 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.520583 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.520856 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.520862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.521173 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:38.521368 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.521373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.522077 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.522078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.522918 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.522920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.523495 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.523496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.523624 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.524167 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:38.524175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:38.525727 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:38.527400 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.527407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.527410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.527640 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.527874 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.527875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.527876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.527988 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.528294 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:38.528300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:38.528429 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.528526 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.528657 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 17:55:38.529448 19727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33357
I20260928 17:55:38.529563 19727 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:38.529773 19727 heartbeater.cc:511] Master 127.0.98.62:33357 requested a full tablet report, sending...
I20260928 17:55:38.530368 19536 ts_manager.cc:194] Registered new tserver with Master: fdb5f7651c15434db726b68968661908 (127.0.98.1:36123)
I20260928 17:55:38.531023 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57137
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 17:55:38.551468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:38.632251 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:38.632532 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:38.632576 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:38.633822 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:38.633908 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:38.633953 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:38.635569 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:38.635865 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:38.636143 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:38.641302 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:38.641311 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.641548 19732 server_base.cc:1034] running on GCE node
W20260928 17:55:38.641316 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.641822 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:38.642088 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:38.643276 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790618138643262 us; error 33 us; skew 500 ppm
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:38.650097 19732 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 17:55:38.650591 19732 webserver.cc:460] Webserver started at http://127.0.98.2:40739/ using document root <none> and password file <none>
I20260928 17:55:38.650934 19732 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:38.651039 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:38.651206 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:38.652362 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e099caad3f242d1a6e8b78de52668ee"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "47c14754a0fd02e0793e9b44f48492fa"
server_key_iv: "2e456f0cc223ed25010df5dd6e567418"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.652745 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e099caad3f242d1a6e8b78de52668ee"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "47c14754a0fd02e0793e9b44f48492fa"
server_key_iv: "2e456f0cc223ed25010df5dd6e567418"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.654131 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:38.654892 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.655056 19732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:38.655154 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "0e099caad3f242d1a6e8b78de52668ee"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "47c14754a0fd02e0793e9b44f48492fa"
server_key_iv: "2e456f0cc223ed25010df5dd6e567418"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.655236 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:38.668888 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:38.671025 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:38.671187 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:38.671425 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:38.671729 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:38.671782 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.671824 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:38.671851 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.677729 19732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42933
I20260928 17:55:38.677832 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42933 every 8 connection(s)
I20260928 17:55:38.678166 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:38.686530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19732
I20260928 17:55:38.686659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:38.686875 392 external_mini_cluster.cc:1428] Setting key 6deb6d7e8ad728ca5314b16edeaeb8d0
I20260928 17:55:38.687587 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.678959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41765 (local address 127.0.98.62:33357)
0928 17:55:38.679037 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.679039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.679139 (+ 100us) server_negotiation.cc:407] Connection header received
0928 17:55:38.680055 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.680056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.680067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.680159 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:38.680758 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.681244 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.682260 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.682441 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.682491 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.684524 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:38.684525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:38.684526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:38.684533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:38.686021 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.686566 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.686567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.686567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.686650 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.686893 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.686894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.686895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.687031 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:38.687113 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.687418 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.687500 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 17:55:38.687776 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.678875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33357 (local address 127.0.98.2:41765)
0928 17:55:38.679026 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.679073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.679966 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.680237 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.680240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.680494 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:38.680687 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.680691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.681345 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.681346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.682181 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.682184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.682512 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.682512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.682612 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.683061 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:38.683069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:38.684425 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:38.686126 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.686134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.686136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.686444 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.686707 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.686708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.686709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.686821 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.687104 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:38.687110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:38.687269 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.687388 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.687541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260928 17:55:38.688316 19861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33357
I20260928 17:55:38.688412 19861 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:38.688606 19861 heartbeater.cc:511] Master 127.0.98.62:33357 requested a full tablet report, sending...
I20260928 17:55:38.688993 19536 ts_manager.cc:194] Registered new tserver with Master: 0e099caad3f242d1a6e8b78de52668ee (127.0.98.2:42933)
I20260928 17:55:38.689370 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41765
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 17:55:38.711421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:38.793401 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:38.793612 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:38.793637 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:38.794730 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:38.794766 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:38.794799 19866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:38.796505 19866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:38.796692 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:38.796909 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:38.801992 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:38.802016 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:38.802016 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:38.802315 19866 server_base.cc:1034] running on GCE node
I20260928 17:55:38.802557 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:38.802815 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:38.803995 19866 hybrid_clock.cc:648] HybridClock initialized: now 1790618138803983 us; error 31 us; skew 500 ppm
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:38.806592 19866 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 17:55:38.807014 19866 webserver.cc:460] Webserver started at http://127.0.98.3:41929/ using document root <none> and password file <none>
I20260928 17:55:38.807216 19866 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:38.807258 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:38.807351 19866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:38.808490 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "1bdee31d778a44529ec1230c965a57f8"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "bd2da44bccca9ac62f4d650703d752b0"
server_key_iv: "f2827713b60b89415cc4cb2375aaea18"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.808800 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1bdee31d778a44529ec1230c965a57f8"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "bd2da44bccca9ac62f4d650703d752b0"
server_key_iv: "f2827713b60b89415cc4cb2375aaea18"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.810130 19866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:38.810927 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.811116 19866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:38.811183 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "1bdee31d778a44529ec1230c965a57f8"
format_stamp: "Formatted at 2026-09-28 17:55:38 on dist-test-slave-zqlt"
server_key: "bd2da44bccca9ac62f4d650703d752b0"
server_key_iv: "f2827713b60b89415cc4cb2375aaea18"
server_key_version: "encryptionkey@0"
I20260928 17:55:38.811241 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:38.827724 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:38.829963 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:38.830082 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:38.830273 19866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:38.830612 19866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:38.830673 19866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.830721 19866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:38.830754 19866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:38.836861 19866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44623
I20260928 17:55:38.836912 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44623 every 8 connection(s)
I20260928 17:55:38.837239 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 17:55:38 dist-test-slave-zqlt krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:38.846585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 19866
I20260928 17:55:38.846668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:38.846884 392 external_mini_cluster.cc:1428] Setting key 97078e61e6e0b0ec05674f2d29fd789a
I20260928 17:55:38.846966 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.838040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59301 (local address 127.0.98.62:33357)
0928 17:55:38.838084 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.838086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.838362 (+ 276us) server_negotiation.cc:407] Connection header received
0928 17:55:38.839356 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.839358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.839372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.839465 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:38.840162 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.840685 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.841719 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.841927 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.842014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.844324 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:38.844325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:38.844326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:38.844334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:38.845536 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.845968 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.845970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.845970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.846033 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.846284 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.846285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.846285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.846407 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:38.846487 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.846786 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.846872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:55:38.847167 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.838038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33357 (local address 127.0.98.3:59301)
0928 17:55:38.838232 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.838267 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.839234 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.839550 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.839554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.839826 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:38.840031 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.840036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.840785 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.840786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.841619 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.841621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.842024 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.842025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.842162 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.842677 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:55:38.842685 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:38.844213 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:38.845647 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.845654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.845657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.845882 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.846107 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:55:38.846107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:55:38.846108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:55:38.846216 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:55:38.846476 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:38.846483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:38.846647 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.846754 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.846924 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260928 17:55:38.847684 19995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33357
I20260928 17:55:38.847788 19995 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:38.847982 19995 heartbeater.cc:511] Master 127.0.98.62:33357 requested a full tablet report, sending...
I20260928 17:55:38.848409 19536 ts_manager.cc:194] Registered new tserver with Master: 1bdee31d778a44529ec1230c965a57f8 (127.0.98.3:44623)
I20260928 17:55:38.848748 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59301
I20260928 17:55:38.858866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:38.866300 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.860516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.0.98.62:33357)
0928 17:55:38.860580 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.860582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.860667 (+ 85us) server_negotiation.cc:407] Connection header received
0928 17:55:38.860751 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.860752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.860760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.860812 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:38.861405 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.861869 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.862589 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.862759 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.862811 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.864129 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:38.864130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:38.864130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:38.864135 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:38.865274 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.865592 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.865593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.865594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.865642 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:38.865815 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:38.865816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:38.865817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:38.865946 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:38.866020 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.866179 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.866238 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 17:55:38.868044 19536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:38.869618 19536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:38.875741 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.871888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36123 (local address 127.0.0.1:39018)
0928 17:55:38.872105 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.872151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.872218 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.872445 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.872449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.872451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:38.872522 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.872524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.873531 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.873532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.874524 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.874527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.875458 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.875459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.875532 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.875545 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.875618 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.875668 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260928 17:55:38.875748 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.872073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.98.1:36123)
0928 17:55:38.872260 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.872266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.872276 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:55:38.872322 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.872324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.872336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.872440 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:38.872586 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.873428 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.874649 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.875371 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.875461 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.875525 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.875634 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.875671 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":118,"threads_started":1}
I20260928 17:55:38.876286 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.871905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42933 (local address 127.0.0.1:57494)
0928 17:55:38.872462 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.872471 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.872506 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.872840 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.872841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.872842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:38.872880 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.872881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.874167 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.874168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.875047 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.875049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.876017 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.876018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.876108 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.876114 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.876162 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.876202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":95,"threads_started":1}
I20260928 17:55:38.876411 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.871888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44623 (local address 127.0.0.1:57876)
0928 17:55:38.872342 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.872351 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.872393 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.872737 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.872738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.872740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:38.872807 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.872809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.874140 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.874141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.875186 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.875189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.876159 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.876160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.876222 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.876226 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.876287 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.876326 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 17:55:38.876470 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.872069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.98.3:44623)
0928 17:55:38.872512 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.872517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.872528 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:55:38.872566 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.872569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.872586 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.872887 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:38.872970 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.874043 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.875279 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.876058 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.876157 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.876221 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.876298 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.876344 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":107,"threads_started":1}
I20260928 17:55:38.876646 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.872296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.0.98.2:42933)
0928 17:55:38.872622 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.872626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.872635 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:55:38.872670 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.872672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.872686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.872753 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:38.872957 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.874027 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.875129 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.875851 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.875924 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.875982 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.876460 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.876511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":87,"threads_started":1}
I20260928 17:55:38.877296 19929 tablet_service.cc:1467] Processing CreateTablet for tablet cb5af450f96348bda399815b8ece1c7e (DEFAULT_TABLE table=test-table [id=8fc1e2ef3f204588af9ed564f74faeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:38.877315 19795 tablet_service.cc:1467] Processing CreateTablet for tablet cb5af450f96348bda399815b8ece1c7e (DEFAULT_TABLE table=test-table [id=8fc1e2ef3f204588af9ed564f74faeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:38.877671 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5af450f96348bda399815b8ece1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:38.877665 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5af450f96348bda399815b8ece1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:38.881037 20013 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Bootstrap starting.
I20260928 17:55:38.881285 19661 tablet_service.cc:1467] Processing CreateTablet for tablet cb5af450f96348bda399815b8ece1c7e (DEFAULT_TABLE table=test-table [id=8fc1e2ef3f204588af9ed564f74faeed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:38.881609 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5af450f96348bda399815b8ece1c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:38.881722 20013 tablet_bootstrap.cc:654] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:38.882220 20013 log.cc:824] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:38.882561 20012 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Bootstrap starting.
I20260928 17:55:38.883005 20013 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: No bootstrap required, opened a new log
I20260928 17:55:38.883080 20013 ts_tablet_manager.cc:1397] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:38.883183 20012 tablet_bootstrap.cc:654] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:38.883863 20012 log.cc:824] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:38.884572 20012 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: No bootstrap required, opened a new log
I20260928 17:55:38.884655 20012 ts_tablet_manager.cc:1397] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:55:38.884944 20016 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: Bootstrap starting.
I20260928 17:55:38.885222 20013 raft_consensus.cc:348] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.885419 20013 raft_consensus.cc:374] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:38.885478 20013 raft_consensus.cc:729] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e099caad3f242d1a6e8b78de52668ee, State: Initialized, Role: FOLLOWER
I20260928 17:55:38.885596 20016 tablet_bootstrap.cc:654] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:38.885632 20013 consensus_queue.cc:260] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.885946 19861 heartbeater.cc:503] Master 127.0.98.62:33357 was elected leader, sending a full tablet report...
I20260928 17:55:38.885950 20013 ts_tablet_manager.cc:1428] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:38.886463 20016 log.cc:824] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:38.886978 20012 raft_consensus.cc:348] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.887161 20012 raft_consensus.cc:374] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:38.887212 20012 raft_consensus.cc:729] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bdee31d778a44529ec1230c965a57f8, State: Initialized, Role: FOLLOWER
I20260928 17:55:38.887336 20012 consensus_queue.cc:260] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.887387 20016 tablet_bootstrap.cc:492] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: No bootstrap required, opened a new log
I20260928 17:55:38.887467 20016 ts_tablet_manager.cc:1397] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 17:55:38.887633 20012 ts_tablet_manager.cc:1428] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 17:55:38.887755 19995 heartbeater.cc:503] Master 127.0.98.62:33357 was elected leader, sending a full tablet report...
I20260928 17:55:38.889313 20016 raft_consensus.cc:348] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.889469 20016 raft_consensus.cc:374] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:38.889516 20016 raft_consensus.cc:729] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb5f7651c15434db726b68968661908, State: Initialized, Role: FOLLOWER
I20260928 17:55:38.889628 20016 consensus_queue.cc:260] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.889876 20016 ts_tablet_manager.cc:1428] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:38.889906 19727 heartbeater.cc:503] Master 127.0.98.62:33357 was elected leader, sending a full tablet report...
I20260928 17:55:38.892961 20020 raft_consensus.cc:482] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:38.893026 20020 raft_consensus.cc:504] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.893258 20020 leader_election.cc:288] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bdee31d778a44529ec1230c965a57f8 (127.0.98.3:44623), 0e099caad3f242d1a6e8b78de52668ee (127.0.98.2:42933)
I20260928 17:55:38.896922 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.893510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37917 (local address 127.0.98.2:42933)
0928 17:55:38.893665 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.893667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.893681 (+ 14us) server_negotiation.cc:407] Connection header received
0928 17:55:38.893740 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.893741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.893752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.893799 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:38.893933 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.894670 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896076 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.896585 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896644 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.896652 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.896813 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.896857 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 17:55:38.896925 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.893472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42933 (local address 127.0.98.1:37917)
0928 17:55:38.893634 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.893643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.893688 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.893807 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.893807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.893808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:38.893863 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.893865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.894793 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.894794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.895984 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.895987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.896659 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.896735 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.896739 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.896789 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.896822 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":57,"thread_start_us":92,"threads_started":1}
I20260928 17:55:38.897009 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.893508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35547 (local address 127.0.98.3:44623)
0928 17:55:38.893674 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.893676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.893684 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:38.893706 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.893707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.893716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.893773 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:38.893933 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.894679 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896110 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.896613 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896701 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.896710 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.896897 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.896937 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260928 17:55:38.897297 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5af450f96348bda399815b8ece1c7e" candidate_uuid: "fdb5f7651c15434db726b68968661908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e099caad3f242d1a6e8b78de52668ee" is_pre_election: true
I20260928 17:55:38.897428 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.893471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44623 (local address 127.0.98.1:35547)
0928 17:55:38.893536 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.893549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:38.893592 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:38.893786 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:38.893788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:38.893790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:38.893861 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.893862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.894750 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.894751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.896003 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:38.896004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.896717 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.896718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:38.896821 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.896825 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:38.896880 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 17:55:38.896920 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":6}
I20260928 17:55:38.897430 19815 raft_consensus.cc:2454] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdb5f7651c15434db726b68968661908 in term 0.
I20260928 17:55:38.897704 19614 leader_election.cc:302] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e099caad3f242d1a6e8b78de52668ee, fdb5f7651c15434db726b68968661908; no voters:
I20260928 17:55:38.897760 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5af450f96348bda399815b8ece1c7e" candidate_uuid: "fdb5f7651c15434db726b68968661908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdee31d778a44529ec1230c965a57f8" is_pre_election: true
I20260928 17:55:38.897821 20020 raft_consensus.cc:2781] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:38.897871 20020 raft_consensus.cc:482] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:38.897923 20020 raft_consensus.cc:3037] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:38.897918 19949 raft_consensus.cc:2454] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdb5f7651c15434db726b68968661908 in term 0.
I20260928 17:55:38.898572 20020 raft_consensus.cc:504] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.898715 20020 leader_election.cc:288] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [CANDIDATE]: Term 1 election: Requested vote from peers 1bdee31d778a44529ec1230c965a57f8 (127.0.98.3:44623), 0e099caad3f242d1a6e8b78de52668ee (127.0.98.2:42933)
I20260928 17:55:38.898880 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5af450f96348bda399815b8ece1c7e" candidate_uuid: "fdb5f7651c15434db726b68968661908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bdee31d778a44529ec1230c965a57f8"
I20260928 17:55:38.898921 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5af450f96348bda399815b8ece1c7e" candidate_uuid: "fdb5f7651c15434db726b68968661908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e099caad3f242d1a6e8b78de52668ee"
I20260928 17:55:38.898948 19949 raft_consensus.cc:3037] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:38.898993 19815 raft_consensus.cc:3037] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:38.899663 19815 raft_consensus.cc:2454] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdb5f7651c15434db726b68968661908 in term 1.
I20260928 17:55:38.899695 19949 raft_consensus.cc:2454] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdb5f7651c15434db726b68968661908 in term 1.
I20260928 17:55:38.899809 19614 leader_election.cc:302] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e099caad3f242d1a6e8b78de52668ee, fdb5f7651c15434db726b68968661908; no voters:
I20260928 17:55:38.899904 20020 raft_consensus.cc:2781] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:38.900065 20020 raft_consensus.cc:686] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [term 1 LEADER]: Becoming Leader. State: Replica: fdb5f7651c15434db726b68968661908, State: Running, Role: LEADER
I20260928 17:55:38.900198 20020 consensus_queue.cc:237] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } }
I20260928 17:55:38.900921 19536 catalog_manager.cc:5495] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdb5f7651c15434db726b68968661908 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fdb5f7651c15434db726b68968661908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdb5f7651c15434db726b68968661908" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 } health_report { overall_health: UNKNOWN } } }
I20260928 17:55:38.910737 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.908554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.0.98.1:36123)
0928 17:55:38.908597 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:38.908599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:38.908630 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:55:38.908688 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:38.908689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:38.908705 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:38.908755 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:38.908854 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.909359 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.910137 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:38.910312 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:38.910362 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:38.910477 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:38.910568 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:38.910622 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:38.910656 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 17:55:38.910671 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:55:38.912432 19949 raft_consensus.cc:1261] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 1 FOLLOWER]: Refusing update from remote peer fdb5f7651c15434db726b68968661908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:38.912433 19815 raft_consensus.cc:1261] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 1 FOLLOWER]: Refusing update from remote peer fdb5f7651c15434db726b68968661908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:38.912819 20022 consensus_queue.cc:1035] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e099caad3f242d1a6e8b78de52668ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:38.912940 20020 consensus_queue.cc:1035] T cb5af450f96348bda399815b8ece1c7e P fdb5f7651c15434db726b68968661908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bdee31d778a44529ec1230c965a57f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:38.915534 20027 mvcc.cc:205] Tried to move back new op lower bound from 7334371896982568960 to 7334371896935784448. Current Snapshot: MvccSnapshot[applied={T|T < 7334371896982568960}]
I20260928 17:55:38.916620 20026 mvcc.cc:205] Tried to move back new op lower bound from 7334371896982568960 to 7334371896935784448. Current Snapshot: MvccSnapshot[applied={T|T < 7334371896982568960 or (T in {7334371896982568960})}]
I20260928 17:55:38.917471 19536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:38.917564 19536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:38.918270 19536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950c4bfb5e7c4ecc8abeef216f16bdb1: Sending DeleteTablet for 3 replicas of tablet cb5af450f96348bda399815b8ece1c7e
I20260928 17:55:38.918581 19929 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5af450f96348bda399815b8ece1c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57876
I20260928 17:55:38.918668 19795 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5af450f96348bda399815b8ece1c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57494
I20260928 17:55:38.918933 20035 tablet_replica.cc:330] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: stopping tablet replica
I20260928 17:55:38.918923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19598
I20260928 17:55:38.918907 19661 tablet_service.cc:1514] Processing DeleteTablet for tablet cb5af450f96348bda399815b8ece1c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39018
I20260928 17:55:38.919026 20035 raft_consensus.cc:2229] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:38.919114 20035 raft_consensus.cc:2258] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:38.919561 20035 ts_tablet_manager.cc:1905] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:38.921319 20035 ts_tablet_manager.cc:1918] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:38.921387 20035 log.cc:1196] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/cb5af450f96348bda399815b8ece1c7e
I20260928 17:55:38.921641 20035 ts_tablet_manager.cc:1939] T cb5af450f96348bda399815b8ece1c7e P 0e099caad3f242d1a6e8b78de52668ee: Deleting consensus metadata
I20260928 17:55:38.921841 20034 tablet_replica.cc:330] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: stopping tablet replica
I20260928 17:55:38.921960 20034 raft_consensus.cc:2229] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:38.922070 20034 raft_consensus.cc:2258] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:38.922323 19520 catalog_manager.cc:4841] TS 0e099caad3f242d1a6e8b78de52668ee (127.0.98.2:42933): tablet cb5af450f96348bda399815b8ece1c7e (table test-table [id=8fc1e2ef3f204588af9ed564f74faeed]) successfully deleted
I20260928 17:55:38.922425 20034 ts_tablet_manager.cc:1905] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:38.923811 20034 ts_tablet_manager.cc:1918] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:38.923871 20034 log.cc:1196] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/cb5af450f96348bda399815b8ece1c7e
I20260928 17:55:38.924181 20034 ts_tablet_manager.cc:1939] T cb5af450f96348bda399815b8ece1c7e P 1bdee31d778a44529ec1230c965a57f8: Deleting consensus metadata
W20260928 17:55:38.924605 19523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36123 (error 108)
I20260928 17:55:38.924639 19522 catalog_manager.cc:4841] TS 1bdee31d778a44529ec1230c965a57f8 (127.0.98.3:44623): tablet cb5af450f96348bda399815b8ece1c7e (table test-table [id=8fc1e2ef3f204588af9ed564f74faeed]) successfully deleted
I20260928 17:55:38.925096 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:38.924959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36123 (local address 127.0.0.1:39034)
0928 17:55:38.925021 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:38.925063 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:55:38.925148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19732
W20260928 17:55:38.925239 19523 catalog_manager.cc:4584] TS fdb5f7651c15434db726b68968661908 (127.0.98.1:36123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb5af450f96348bda399815b8ece1c7e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36123: connect: Connection refused (error 111)
I20260928 17:55:38.930399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19866
I20260928 17:55:38.934989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 19504
2026-09-28T17:55:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T17:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:38Z Disabled control of system clock
I20260928 17:55:38.962994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:39.043571 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:39.043818 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:39.043839 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:39.044989 20040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:39.045028 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:39.045042 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:39.045054 20040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:39.045068 20040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:39.046377 20040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:39.046593 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:39.046877 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:39.051798 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:39.051849 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.051896 20040 server_base.cc:1034] running on GCE node
W20260928 17:55:39.051858 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.052263 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:39.052628 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:39.053802 20040 hybrid_clock.cc:648] HybridClock initialized: now 1790618139053769 us; error 38 us; skew 500 ppm
I20260928 17:55:39.055447 20040 webserver.cc:460] Webserver started at http://127.0.98.62:44761/ using document root <none> and password file <none>
I20260928 17:55:39.055807 20040 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:39.055892 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:39.056039 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:39.057874 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "2c43fad026ec4ef68dfffd1e8596d75a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "c923502cd4dd2f95b8580917497ece19"
server_key_iv: "59d247ef19b5f860cb23f6203ce5b163"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.058404 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c43fad026ec4ef68dfffd1e8596d75a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "c923502cd4dd2f95b8580917497ece19"
server_key_iv: "59d247ef19b5f860cb23f6203ce5b163"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.059886 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 17:55:39.060870 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.061053 20040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:39.061165 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "2c43fad026ec4ef68dfffd1e8596d75a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "c923502cd4dd2f95b8580917497ece19"
server_key_iv: "59d247ef19b5f860cb23f6203ce5b163"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.061276 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:39.069936 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:39.070233 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:39.070391 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:39.074386 20040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34223
I20260928 17:55:39.074435 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34223 every 8 connection(s)
I20260928 17:55:39.074786 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:39.075969 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:39.077455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20040
I20260928 17:55:39.077565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:39.077775 392 external_mini_cluster.cc:1428] Setting key e3097a06fef705bf9272233d6354e433
I20260928 17:55:39.081001 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: Bootstrap starting.
I20260928 17:55:39.081686 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:39.082010 20107 log.cc:824] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:39.082801 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: No bootstrap required, opened a new log
I20260928 17:55:39.085444 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } }
I20260928 17:55:39.085664 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:39.085707 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c43fad026ec4ef68dfffd1e8596d75a, State: Initialized, Role: FOLLOWER
I20260928 17:55:39.085863 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } }
I20260928 17:55:39.085958 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:39.085990 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:39.086036 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:39.086755 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } }
I20260928 17:55:39.086870 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c43fad026ec4ef68dfffd1e8596d75a; no voters:
I20260928 17:55:39.087383 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:39.087432 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:39.087538 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [term 1 LEADER]: Becoming Leader. State: Replica: 2c43fad026ec4ef68dfffd1e8596d75a, State: Running, Role: LEADER
I20260928 17:55:39.087618 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } }
I20260928 17:55:39.087661 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:39.087917 20112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c43fad026ec4ef68dfffd1e8596d75a. Latest consensus state: current_term: 1 leader_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } } }
I20260928 17:55:39.087988 20112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:39.087927 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c43fad026ec4ef68dfffd1e8596d75a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34223 } } }
I20260928 17:55:39.088042 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:39.088212 20116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:39.088773 20116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:39.091997 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.078469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.62:34223)
0928 17:55:39.078670 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.078680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.082443 (+ 3763us) server_negotiation.cc:407] Connection header received
0928 17:55:39.085941 (+ 3498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.085950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.086027 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.086149 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:39.086642 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.088019 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.089364 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.089976 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.090066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.091061 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:39.091067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:39.091067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:39.091289 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:39.091337 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:39.091354 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:39.091440 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.091658 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.091748 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 17:55:39.091828 20129 mvcc.cc:205] Tried to move back new op lower bound from 7334371897715994624 to 7334371897703206912. Current Snapshot: MvccSnapshot[applied={T|T < 7334371897715994624}]
I20260928 17:55:39.092630 20116 catalog_manager.cc:1347] Generated new cluster ID: ab44a8d3967c489f8e73f291ead570e5
I20260928 17:55:39.092721 20116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:39.106251 20116 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:39.106976 20116 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:39.112972 20116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: Generated new TSK 0
I20260928 17:55:39.113157 20116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:39.119660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--builtin_ntp_servers=127.0.98.20:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:39.203471 20131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:39.203680 20131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:39.203702 20131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:39.204946 20131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:39.204988 20131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:39.205013 20131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 17:55:39.206363 20131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:39.206617 20131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:39.206945 20131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:39.212007 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:39.212018 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.212239 20131 server_base.cc:1034] running on GCE node
W20260928 17:55:39.212009 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.212646 20131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:39.212960 20131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:39.214134 20131 hybrid_clock.cc:648] HybridClock initialized: now 1790618139214121 us; error 29 us; skew 500 ppm
I20260928 17:55:39.215663 20131 webserver.cc:460] Webserver started at http://127.0.98.1:38403/ using document root <none> and password file <none>
I20260928 17:55:39.215987 20131 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:39.216059 20131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:39.216233 20131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:39.217377 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae036ba2f5ac43a4b59a391c55430c41"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "1625ca069a23ec16d262ad9c5baa6e65"
server_key_iv: "7f1358206d48edf7c774326f6a4ba5e9"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.217681 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae036ba2f5ac43a4b59a391c55430c41"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "1625ca069a23ec16d262ad9c5baa6e65"
server_key_iv: "7f1358206d48edf7c774326f6a4ba5e9"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.218978 20131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:55:39.219694 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.219897 20131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:39.219996 20131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal
uuid: "ae036ba2f5ac43a4b59a391c55430c41"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "1625ca069a23ec16d262ad9c5baa6e65"
server_key_iv: "7f1358206d48edf7c774326f6a4ba5e9"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.220065 20131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:39.238831 20131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:39.239135 20131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:39.239240 20131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:39.239445 20131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:39.239802 20131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:39.239840 20131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.239863 20131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:39.239910 20131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.245692 20131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38693
I20260928 17:55:39.245836 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38693 every 8 connection(s)
I20260928 17:55:39.246138 20131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/data/info.pb
I20260928 17:55:39.251787 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.246979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42149 (local address 127.0.98.62:34223)
0928 17:55:39.247086 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.247089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.247299 (+ 210us) server_negotiation.cc:407] Connection header received
0928 17:55:39.247816 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.247817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.247834 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.247900 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:39.248686 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.249240 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.250245 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.250453 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.250521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.251410 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:39.251411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:39.251412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:39.251416 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:39.251438 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:39.251443 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:39.251496 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.251604 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.251668 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 17:55:39.252002 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.246922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34223 (local address 127.0.98.1:42149)
0928 17:55:39.247165 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.247202 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.247687 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.247996 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.247999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.248008 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:39.248567 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.248572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.249329 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.249329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.250141 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.250143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.250538 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.250538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.250653 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.251255 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:39.251256 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:39.251288 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:39.251314 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:39.251510 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:39.251515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:39.251517 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.251573 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.251716 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":130,"threads_started":1}
I20260928 17:55:39.252709 20259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34223
I20260928 17:55:39.252857 20259 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:39.253105 20259 heartbeater.cc:511] Master 127.0.98.62:34223 requested a full tablet report, sending...
I20260928 17:55:39.253618 20071 ts_manager.cc:194] Registered new tserver with Master: ae036ba2f5ac43a4b59a391c55430c41 (127.0.98.1:38693)
I20260928 17:55:39.254197 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42149
I20260928 17:55:39.255617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20131
I20260928 17:55:39.255745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-0/wal/instance
I20260928 17:55:39.255949 392 external_mini_cluster.cc:1428] Setting key 3c0fe02cb009c63cf84887b67180444f
I20260928 17:55:39.256847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--builtin_ntp_servers=127.0.98.20:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 17:55:39.338223 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:39.338433 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:39.338454 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:39.339633 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:39.339677 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:39.339702 20262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 17:55:39.341096 20262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:39.341358 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:39.341656 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:39.346700 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:39.346740 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.346771 20262 server_base.cc:1034] running on GCE node
W20260928 17:55:39.346714 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.347123 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:39.347309 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:39.348443 20262 hybrid_clock.cc:648] HybridClock initialized: now 1790618139348429 us; error 29 us; skew 500 ppm
I20260928 17:55:39.349670 20262 webserver.cc:460] Webserver started at http://127.0.98.2:44571/ using document root <none> and password file <none>
I20260928 17:55:39.349910 20262 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:39.349957 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:39.350127 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:39.351176 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data/instance:
uuid: "1797356eeb0244a9a69f415b0368b41a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "9a391d8c95ab9cbb07ae998de909fae2"
server_key_iv: "458ad7e2966f7325989da9cb57d8466d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.351534 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1797356eeb0244a9a69f415b0368b41a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "9a391d8c95ab9cbb07ae998de909fae2"
server_key_iv: "458ad7e2966f7325989da9cb57d8466d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.352914 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:39.353603 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.353770 20262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:39.353866 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal
uuid: "1797356eeb0244a9a69f415b0368b41a"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "9a391d8c95ab9cbb07ae998de909fae2"
server_key_iv: "458ad7e2966f7325989da9cb57d8466d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.353950 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:39.391494 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:39.391867 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:39.392037 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:39.392344 20262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:39.392709 20262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:39.392778 20262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.392823 20262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:39.392851 20262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.398645 20262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38159
I20260928 17:55:39.398692 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38159 every 8 connection(s)
I20260928 17:55:39.399049 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/data/info.pb
I20260928 17:55:39.402014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20262
I20260928 17:55:39.402122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal/instance
I20260928 17:55:39.402320 392 external_mini_cluster.cc:1428] Setting key b01337a6bf81b6912d84b3a7c323d0c8
I20260928 17:55:39.403316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--builtin_ntp_servers=127.0.98.20:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 17:55:39.405033 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.399797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52287 (local address 127.0.98.62:34223)
0928 17:55:39.399848 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.399851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.400148 (+ 297us) server_negotiation.cc:407] Connection header received
0928 17:55:39.400744 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.400745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.400759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.400826 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:39.401551 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.402260 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.403345 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.403647 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.403727 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.404737 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:39.404739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:39.404739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:39.404744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:39.404767 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:39.404772 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:39.404839 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.404923 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.404954 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:55:39.405264 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.399781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34223 (local address 127.0.98.2:52287)
0928 17:55:39.399991 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.400028 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.400622 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.400906 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.400911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.400922 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:39.401448 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.401453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.402393 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.402396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.403253 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.403256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.403758 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.403759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.403881 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.404581 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:39.404582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:39.404629 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:39.404661 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:39.404841 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:39.404847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:39.404850 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.404925 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.405033 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260928 17:55:39.405872 20390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34223
I20260928 17:55:39.405972 20390 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:39.406172 20390 heartbeater.cc:511] Master 127.0.98.62:34223 requested a full tablet report, sending...
I20260928 17:55:39.406550 20071 ts_manager.cc:194] Registered new tserver with Master: 1797356eeb0244a9a69f415b0368b41a (127.0.98.2:38159)
I20260928 17:55:39.407003 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52287
W20260928 17:55:39.486155 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:39.486359 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:39.486380 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:39.487588 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:39.487627 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:39.487651 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 17:55:39.489176 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:39.489444 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:39.489743 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:39.494848 20401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:39.494829 20398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:39.494812 20399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:39.495021 20393 server_base.cc:1034] running on GCE node
I20260928 17:55:39.495234 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:39.495469 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:39.496631 20393 hybrid_clock.cc:648] HybridClock initialized: now 1790618139496624 us; error 38 us; skew 500 ppm
I20260928 17:55:39.497928 20393 webserver.cc:460] Webserver started at http://127.0.98.3:42571/ using document root <none> and password file <none>
I20260928 17:55:39.498180 20393 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:39.498250 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:39.498406 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:39.499444 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data/instance:
uuid: "98b800ddf5634bb8af7a07d6e9d44542"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "62f9aa4daab474e44e472f40bda1034f"
server_key_iv: "b3ea664c809c98e4fc7fe4e669f1635d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.499797 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98b800ddf5634bb8af7a07d6e9d44542"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "62f9aa4daab474e44e472f40bda1034f"
server_key_iv: "b3ea664c809c98e4fc7fe4e669f1635d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.501129 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:55:39.501820 20407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.501998 20393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:39.502095 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal
uuid: "98b800ddf5634bb8af7a07d6e9d44542"
format_stamp: "Formatted at 2026-09-28 17:55:39 on dist-test-slave-zqlt"
server_key: "62f9aa4daab474e44e472f40bda1034f"
server_key_iv: "b3ea664c809c98e4fc7fe4e669f1635d"
server_key_version: "encryptionkey@0"
I20260928 17:55:39.502177 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:39.515277 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:39.515583 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:39.515748 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:39.515975 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 17:55:39.516351 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 17:55:39.516410 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.516445 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 17:55:39.516474 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:39.522413 20393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36305
I20260928 17:55:39.522466 20520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36305 every 8 connection(s)
I20260928 17:55:39.522795 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/data/info.pb
I20260928 17:55:39.528038 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.523555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51697 (local address 127.0.98.62:34223)
0928 17:55:39.523623 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.523625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.523873 (+ 248us) server_negotiation.cc:407] Connection header received
0928 17:55:39.524457 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.524458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.524467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.524525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:39.525172 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.525702 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.526683 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.526879 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.526937 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.527791 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:39.527792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:39.527793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:39.527796 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:39.527806 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:39.527810 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:39.527854 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.527952 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.527977 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:55:39.528288 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.523535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34223 (local address 127.0.98.3:51697)
0928 17:55:39.523740 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.523783 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.524360 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.524591 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.524595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.524603 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:55:39.525101 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.525105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.525787 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.525788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.526582 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.526584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.526958 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.526959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.527067 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.527660 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 17:55:39.527661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:55:39.527692 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 17:55:39.527714 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:55:39.527879 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:55:39.527882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:55:39.527885 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.527939 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.528024 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260928 17:55:39.528833 20521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34223
I20260928 17:55:39.528968 20521 heartbeater.cc:464] Registering TS with master...
I20260928 17:55:39.528975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20393
I20260928 17:55:39.529073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal/instance
I20260928 17:55:39.529204 20521 heartbeater.cc:511] Master 127.0.98.62:34223 requested a full tablet report, sending...
I20260928 17:55:39.529314 392 external_mini_cluster.cc:1428] Setting key 48d38067809e5ece646d056a978b2965
I20260928 17:55:39.529661 20071 ts_manager.cc:194] Registered new tserver with Master: 98b800ddf5634bb8af7a07d6e9d44542 (127.0.98.3:36305)
I20260928 17:55:39.530165 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51697
I20260928 17:55:39.530216 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 17:55:39.534437 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.531759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.0.98.62:34223)
0928 17:55:39.531819 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.531820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.531932 (+ 112us) server_negotiation.cc:407] Connection header received
0928 17:55:39.531971 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.531972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.531981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.532029 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:39.532241 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.532691 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.533404 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.533579 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.533632 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.534279 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:39.534280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:39.534280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:39.534283 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:39.534293 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:39.534296 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:39.534333 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.534368 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.534386 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 17:55:39.536060 20071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 17:55:39.537648 20071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 17:55:39.543792 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.540154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.0.98.2:38159)
0928 17:55:39.540355 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.540359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.540595 (+ 236us) server_negotiation.cc:407] Connection header received
0928 17:55:39.540647 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.540649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.540665 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.540736 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:39.540928 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541619 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.542717 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.543406 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.543482 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.543571 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.543672 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.543711 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 17:55:39.543804 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.540027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38159 (local address 127.0.0.1:38324)
0928 17:55:39.540337 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.540347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.540398 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.540806 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.540807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.540808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:39.540851 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.540852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541708 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.541709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.542641 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.542644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.543495 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.543496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.543576 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.543589 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.543660 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.543710 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":99,"threads_started":1}
I20260928 17:55:39.544346 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.539632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38693 (local address 127.0.0.1:60462)
0928 17:55:39.540180 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.540236 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.540354 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.540550 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.540552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.540554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:39.540621 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.540622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541974 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.541975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.543164 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.543166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.544127 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.544128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.544193 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.544196 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.544258 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.544298 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":46,"thread_start_us":73,"threads_started":1}
I20260928 17:55:39.544413 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.539768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.98.1:38693)
0928 17:55:39.539976 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.539982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.540321 (+ 339us) server_negotiation.cc:407] Connection header received
0928 17:55:39.540435 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.540438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.540453 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.540532 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:39.540713 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541876 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.543251 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.544002 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.544112 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.544197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.544273 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.544313 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
I20260928 17:55:39.544942 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.539841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36305 (local address 127.0.0.1:56456)
0928 17:55:39.540181 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.540224 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.540310 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.540680 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.540681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.540683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:39.540754 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.540756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541972 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.541973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.543201 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.543204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.544713 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.544714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.544789 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.544793 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.544847 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.544885 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":36,"thread_start_us":71,"threads_started":1}
I20260928 17:55:39.544850 20324 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4c15af02294735bdc2984d2a637e69 (DEFAULT_TABLE table=test-table [id=8263d75bef544014bf80cfe5321479e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:39.545220 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.539966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.98.3:36305)
0928 17:55:39.540150 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.540155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.540480 (+ 325us) server_negotiation.cc:407] Connection header received
0928 17:55:39.540524 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.540527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.540544 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.540615 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:39.540831 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.541876 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.543293 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.543998 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.544094 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.544198 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.545090 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.545131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260928 17:55:39.545121 20193 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4c15af02294735bdc2984d2a637e69 (DEFAULT_TABLE table=test-table [id=8263d75bef544014bf80cfe5321479e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:39.545239 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4c15af02294735bdc2984d2a637e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:39.545493 20193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4c15af02294735bdc2984d2a637e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:39.545830 20455 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4c15af02294735bdc2984d2a637e69 (DEFAULT_TABLE table=test-table [id=8263d75bef544014bf80cfe5321479e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 17:55:39.546164 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4c15af02294735bdc2984d2a637e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:39.549301 20536 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: Bootstrap starting.
I20260928 17:55:39.549412 20535 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Bootstrap starting.
I20260928 17:55:39.549978 20536 tablet_bootstrap.cc:654] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:39.550100 20537 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Bootstrap starting.
I20260928 17:55:39.550210 20535 tablet_bootstrap.cc:654] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:39.550311 20536 log.cc:824] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:39.550632 20535 log.cc:824] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:39.550762 20537 tablet_bootstrap.cc:654] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:39.551098 20537 log.cc:824] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:39.551362 20536 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: No bootstrap required, opened a new log
I20260928 17:55:39.551442 20535 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: No bootstrap required, opened a new log
I20260928 17:55:39.551447 20536 ts_tablet_manager.cc:1397] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:55:39.551525 20535 ts_tablet_manager.cc:1397] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 17:55:39.551854 20537 tablet_bootstrap.cc:492] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: No bootstrap required, opened a new log
I20260928 17:55:39.551939 20537 ts_tablet_manager.cc:1397] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 17:55:39.553363 20536 raft_consensus.cc:348] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.553519 20536 raft_consensus.cc:374] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:39.553563 20536 raft_consensus.cc:729] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae036ba2f5ac43a4b59a391c55430c41, State: Initialized, Role: FOLLOWER
I20260928 17:55:39.553706 20536 consensus_queue.cc:260] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.553959 20536 ts_tablet_manager.cc:1428] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 17:55:39.554103 20259 heartbeater.cc:503] Master 127.0.98.62:34223 was elected leader, sending a full tablet report...
I20260928 17:55:39.554592 20535 raft_consensus.cc:348] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.554795 20535 raft_consensus.cc:374] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:39.554849 20535 raft_consensus.cc:729] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1797356eeb0244a9a69f415b0368b41a, State: Initialized, Role: FOLLOWER
I20260928 17:55:39.554903 20537 raft_consensus.cc:348] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.555071 20537 raft_consensus.cc:374] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:39.555119 20537 raft_consensus.cc:729] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b800ddf5634bb8af7a07d6e9d44542, State: Initialized, Role: FOLLOWER
I20260928 17:55:39.555079 20535 consensus_queue.cc:260] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.555246 20537 consensus_queue.cc:260] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.555362 20535 ts_tablet_manager.cc:1428] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 17:55:39.555526 20537 ts_tablet_manager.cc:1428] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 17:55:39.555529 20390 heartbeater.cc:503] Master 127.0.98.62:34223 was elected leader, sending a full tablet report...
I20260928 17:55:39.555581 20521 heartbeater.cc:503] Master 127.0.98.62:34223 was elected leader, sending a full tablet report...
I20260928 17:55:39.596460 20542 raft_consensus.cc:482] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:39.596582 20542 raft_consensus.cc:504] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.596872 20542 leader_election.cc:288] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae036ba2f5ac43a4b59a391c55430c41 (127.0.98.1:38693), 98b800ddf5634bb8af7a07d6e9d44542 (127.0.98.3:36305)
I20260928 17:55:39.599699 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.597043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38693 (local address 127.0.98.2:55969)
0928 17:55:39.597097 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.597110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.597152 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.597382 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.597384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.597386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:39.597467 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.597469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.598167 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.598168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.598908 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.598910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.599441 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.599511 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.599514 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.599588 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.599629 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:55:39.599697 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.597127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55969 (local address 127.0.98.1:38693)
0928 17:55:39.597191 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.597193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.597205 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:55:39.597234 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.597235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.597250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.597344 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:39.597544 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.598080 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599006 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.599356 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599442 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.599452 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.599592 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.599629 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 17:55:39.599838 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.597270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43325 (local address 127.0.98.3:36305)
0928 17:55:39.597344 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.597347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.597399 (+ 52us) server_negotiation.cc:407] Connection header received
0928 17:55:39.597470 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.597471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.597486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.597555 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 17:55:39.597737 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.598217 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599136 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.599496 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599574 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.599583 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.599738 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.599772 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 17:55:39.599852 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.597172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36305 (local address 127.0.98.2:43325)
0928 17:55:39.597327 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.597343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 17:55:39.597410 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:55:39.597557 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:55:39.597558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:55:39.597559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 17:55:39.597666 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.597668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.598322 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.598323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.599050 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:55:39.599051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.599588 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.599590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:55:39.599667 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.599670 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 17:55:39.599734 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:55:39.599769 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260928 17:55:39.600039 20213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4c15af02294735bdc2984d2a637e69" candidate_uuid: "1797356eeb0244a9a69f415b0368b41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae036ba2f5ac43a4b59a391c55430c41" is_pre_election: true
I20260928 17:55:39.600221 20213 raft_consensus.cc:2454] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1797356eeb0244a9a69f415b0368b41a in term 0.
I20260928 17:55:39.600489 20280 leader_election.cc:302] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1797356eeb0244a9a69f415b0368b41a, ae036ba2f5ac43a4b59a391c55430c41; no voters:
I20260928 17:55:39.600627 20542 raft_consensus.cc:2781] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 17:55:39.600612 20475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4c15af02294735bdc2984d2a637e69" candidate_uuid: "1797356eeb0244a9a69f415b0368b41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b800ddf5634bb8af7a07d6e9d44542" is_pre_election: true
I20260928 17:55:39.600700 20542 raft_consensus.cc:482] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 17:55:39.600737 20542 raft_consensus.cc:3037] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:39.600764 20475 raft_consensus.cc:2454] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1797356eeb0244a9a69f415b0368b41a in term 0.
I20260928 17:55:39.601567 20542 raft_consensus.cc:504] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.601696 20542 leader_election.cc:288] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [CANDIDATE]: Term 1 election: Requested vote from peers ae036ba2f5ac43a4b59a391c55430c41 (127.0.98.1:38693), 98b800ddf5634bb8af7a07d6e9d44542 (127.0.98.3:36305)
I20260928 17:55:39.601872 20475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4c15af02294735bdc2984d2a637e69" candidate_uuid: "1797356eeb0244a9a69f415b0368b41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98b800ddf5634bb8af7a07d6e9d44542"
I20260928 17:55:39.601955 20475 raft_consensus.cc:3037] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:39.601939 20213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4c15af02294735bdc2984d2a637e69" candidate_uuid: "1797356eeb0244a9a69f415b0368b41a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae036ba2f5ac43a4b59a391c55430c41"
I20260928 17:55:39.602028 20213 raft_consensus.cc:3037] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:39.602671 20475 raft_consensus.cc:2454] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1797356eeb0244a9a69f415b0368b41a in term 1.
I20260928 17:55:39.602698 20213 raft_consensus.cc:2454] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1797356eeb0244a9a69f415b0368b41a in term 1.
I20260928 17:55:39.602876 20280 leader_election.cc:302] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1797356eeb0244a9a69f415b0368b41a, ae036ba2f5ac43a4b59a391c55430c41; no voters:
I20260928 17:55:39.602981 20542 raft_consensus.cc:2781] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:39.603121 20542 raft_consensus.cc:686] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 1 LEADER]: Becoming Leader. State: Replica: 1797356eeb0244a9a69f415b0368b41a, State: Running, Role: LEADER
I20260928 17:55:39.603222 20542 consensus_queue.cc:237] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } }
I20260928 17:55:39.604146 20070 catalog_manager.cc:5495] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1797356eeb0244a9a69f415b0368b41a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1797356eeb0244a9a69f415b0368b41a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1797356eeb0244a9a69f415b0368b41a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38159 } health_report { overall_health: HEALTHY } } }
I20260928 17:55:39.620013 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.617970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.0.98.2:38159)
0928 17:55:39.618031 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:39.618033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:39.618041 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:39.618069 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:39.618069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:39.618085 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:39.618144 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:55:39.618237 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.618675 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.619358 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:39.619531 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:39.619579 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:39.619693 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:55:39.619804 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:55:39.619860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:39.619890 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 17:55:39.619906 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:55:39.621695 20475 raft_consensus.cc:1261] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 1 FOLLOWER]: Refusing update from remote peer 1797356eeb0244a9a69f415b0368b41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:39.621749 20213 raft_consensus.cc:1261] T 3c4c15af02294735bdc2984d2a637e69 P ae036ba2f5ac43a4b59a391c55430c41 [term 1 FOLLOWER]: Refusing update from remote peer 1797356eeb0244a9a69f415b0368b41a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:55:39.622135 20545 consensus_queue.cc:1035] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae036ba2f5ac43a4b59a391c55430c41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:39.622259 20542 consensus_queue.cc:1035] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [LEADER]: Connected to new peer: Peer: permanent_uuid: "98b800ddf5634bb8af7a07d6e9d44542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:55:39.625219 20550 mvcc.cc:205] Tried to move back new op lower bound from 7334371899887874048 to 7334371899815563264. Current Snapshot: MvccSnapshot[applied={T|T < 7334371899887874048}]
I20260928 17:55:39.625900 20549 mvcc.cc:205] Tried to move back new op lower bound from 7334371899887874048 to 7334371899815563264. Current Snapshot: MvccSnapshot[applied={T|T < 7334371899887874048}]
I20260928 17:55:39.627919 20070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:39.628015 20070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 17:55:39.628803 20070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c43fad026ec4ef68dfffd1e8596d75a: Sending DeleteTablet for 3 replicas of tablet 3c4c15af02294735bdc2984d2a637e69
I20260928 17:55:39.629139 20193 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4c15af02294735bdc2984d2a637e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:39 UTC) from {username='slave'} at 127.0.0.1:60462
I20260928 17:55:39.629247 20455 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4c15af02294735bdc2984d2a637e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:39 UTC) from {username='slave'} at 127.0.0.1:56456
I20260928 17:55:39.629462 20559 tablet_replica.cc:330] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: stopping tablet replica
I20260928 17:55:39.629494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20131
I20260928 17:55:39.629555 20559 raft_consensus.cc:2229] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 17:55:39.629657 20559 raft_consensus.cc:2258] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:39.629657 20324 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4c15af02294735bdc2984d2a637e69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 17:55:39 UTC) from {username='slave'} at 127.0.0.1:38324
I20260928 17:55:39.629951 20560 tablet_replica.cc:330] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: stopping tablet replica
I20260928 17:55:39.630076 20560 raft_consensus.cc:2229] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 1 LEADER]: Raft consensus shutting down.
I20260928 17:55:39.630247 20560 raft_consensus.cc:2258] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 17:55:39.630725 20560 ts_tablet_manager.cc:1905] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:39.632509 20560 ts_tablet_manager.cc:1918] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:39.632606 20560 log.cc:1196] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-1/wal/wals/3c4c15af02294735bdc2984d2a637e69
I20260928 17:55:39.632853 20560 ts_tablet_manager.cc:1939] T 3c4c15af02294735bdc2984d2a637e69 P 1797356eeb0244a9a69f415b0368b41a: Deleting consensus metadata
I20260928 17:55:39.633214 20058 catalog_manager.cc:4841] TS 1797356eeb0244a9a69f415b0368b41a (127.0.98.2:38159): tablet 3c4c15af02294735bdc2984d2a637e69 (table test-table [id=8263d75bef544014bf80cfe5321479e5]) successfully deleted
I20260928 17:55:39.633849 20559 ts_tablet_manager.cc:1905] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 17:55:39.635174 20559 ts_tablet_manager.cc:1918] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 17:55:39.635257 20559 log.cc:1196] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Deleting WAL directory at /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790617898245922-392-0/minicluster-data/ts-2/wal/wals/3c4c15af02294735bdc2984d2a637e69
I20260928 17:55:39.635474 20559 ts_tablet_manager.cc:1939] T 3c4c15af02294735bdc2984d2a637e69 P 98b800ddf5634bb8af7a07d6e9d44542: Deleting consensus metadata
I20260928 17:55:39.635870 20058 catalog_manager.cc:4841] TS 98b800ddf5634bb8af7a07d6e9d44542 (127.0.98.3:36305): tablet 3c4c15af02294735bdc2984d2a637e69 (table test-table [id=8263d75bef544014bf80cfe5321479e5]) successfully deleted
W20260928 17:55:39.637748 20058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38693 (error 108)
I20260928 17:55:39.637940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20262
I20260928 17:55:39.640478 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:39.640318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38693 (local address 127.0.0.1:60474)
0928 17:55:39.640378 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 17:55:39.640437 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 17:55:39.640626 20058 catalog_manager.cc:4584] TS ae036ba2f5ac43a4b59a391c55430c41 (127.0.98.1:38693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c4c15af02294735bdc2984d2a637e69: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38693: connect: Connection refused (error 111)
I20260928 17:55:39.643630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20393
I20260928 17:55:39.648018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20040
2026-09-28T17:55:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (712 ms)
[----------] 6 tests from EncryptionPolicyTest (8143 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:39 dist-test-slave-zqlt krb5kdc[20564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:39 dist-test-slave-zqlt krb5kdc[20564](info): set up 2 sockets
Sep 28 17:55:39 dist-test-slave-zqlt krb5kdc[20564](info): commencing operation
krb5kdc: starting...
W20260928 17:55:40.688078 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:40 dist-test-slave-zqlt krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:40.803089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:40.883807 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:40.884054 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:40.884091 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:40.885273 20577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:40.885346 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:40.885372 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:40.885411 20577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:40.885439 20577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:40.886932 20577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:40.887223 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:40.887476 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:40.892525 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:40.892529 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:40.892549 20577 server_base.cc:1034] running on GCE node
W20260928 17:55:40.892549 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:40.893124 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:40.893453 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:40.894608 20577 hybrid_clock.cc:648] HybridClock initialized: now 1790618140894586 us; error 37 us; skew 500 ppm
Sep 28 17:55:40 dist-test-slave-zqlt krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:40.897389 20577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:40.897912 20577 webserver.cc:460] Webserver started at http://127.0.0.1:37011/ using document root <none> and password file <none>
I20260928 17:55:40.898185 20577 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:40.898253 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:40.898399 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:40.899513 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "e2eb95e48a7a4dba913dae4d07ff7c87"
format_stamp: "Formatted at 2026-09-28 17:55:40 on dist-test-slave-zqlt"
server_key: "502f4a732970321079e95b213cd17309"
server_key_iv: "f7058ea7c62742edb951857750c67c81"
server_key_version: "encryptionkey@0"
I20260928 17:55:40.899920 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2eb95e48a7a4dba913dae4d07ff7c87"
format_stamp: "Formatted at 2026-09-28 17:55:40 on dist-test-slave-zqlt"
server_key: "502f4a732970321079e95b213cd17309"
server_key_iv: "f7058ea7c62742edb951857750c67c81"
server_key_version: "encryptionkey@0"
I20260928 17:55:40.901314 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:40.902071 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:40.902257 20577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:40.902362 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "e2eb95e48a7a4dba913dae4d07ff7c87"
format_stamp: "Formatted at 2026-09-28 17:55:40 on dist-test-slave-zqlt"
server_key: "502f4a732970321079e95b213cd17309"
server_key_iv: "f7058ea7c62742edb951857750c67c81"
server_key_version: "encryptionkey@0"
I20260928 17:55:40.902447 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:40.920847 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:40.923345 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:40.923560 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:40.927762 20577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36041
I20260928 17:55:40.927799 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36041 every 8 connection(s)
I20260928 17:55:40.928187 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:40.929039 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:40.931445 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87: Bootstrap starting.
I20260928 17:55:40.932050 20645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:40.932391 20645 log.cc:824] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:40.933112 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87: No bootstrap required, opened a new log
I20260928 17:55:40.934762 20645 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260928 17:55:40.934942 20645 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:40.934993 20645 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2eb95e48a7a4dba913dae4d07ff7c87, State: Initialized, Role: FOLLOWER
I20260928 17:55:40.935135 20645 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260928 17:55:40.935220 20645 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:40.935264 20645 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:40.935328 20645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:40.935891 20645 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260928 17:55:40.936022 20645 leader_election.cc:302] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2eb95e48a7a4dba913dae4d07ff7c87; no voters:
I20260928 17:55:40.936233 20645 leader_election.cc:288] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:40.936296 20648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:40.936486 20648 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [term 1 LEADER]: Becoming Leader. State: Replica: e2eb95e48a7a4dba913dae4d07ff7c87, State: Running, Role: LEADER
I20260928 17:55:40.936522 20645 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:40.936578 20648 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20260928 17:55:40.937171 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2eb95e48a7a4dba913dae4d07ff7c87. Latest consensus state: current_term: 1 leader_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260928 17:55:40.937270 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:40.937455 20649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2eb95e48a7a4dba913dae4d07ff7c87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20260928 17:55:40.937527 20649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:40.937637 20657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:40.937929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20577
I20260928 17:55:40.938062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:40.938146 20657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:40.938268 392 external_mini_cluster.cc:1428] Setting key 7a056059035a183a53c3710b16fb5923
I20260928 17:55:40.941102 20657 catalog_manager.cc:1347] Generated new cluster ID: 67bc76e271314fc8ba16646cf4401c9a
I20260928 17:55:40.941157 20657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:40 dist-test-slave-zqlt krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:55:40.951671 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:40.938753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35751 (local address 127.0.0.1:36041)
0928 17:55:40.940533 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:40.940542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:40.940574 (+ 32us) server_negotiation.cc:407] Connection header received
0928 17:55:40.940680 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:40.940686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:40.940729 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:40.940873 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:40.941682 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:40.943157 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:40.944225 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:40.944960 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:40.945077 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:40.947905 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:40.947909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:40.947910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:40.947923 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:40.949929 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:40.950396 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:40.950399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:40.950399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:40.950509 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:40.950752 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:40.950753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:40.950753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:40.951055 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:40.951201 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:40.951339 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 17:55:40.951443 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":78,"threads_started":1}
I20260928 17:55:40.952719 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:40.960852 20657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:40.961481 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:40.954144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42513 (local address 127.0.0.1:36041)
0928 17:55:40.954177 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:40.954178 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:40.954293 (+ 115us) server_negotiation.cc:407] Connection header received
0928 17:55:40.954352 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:40.954353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:40.954364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:40.954431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:40.955050 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:40.955789 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:40.956821 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:40.957128 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:40.957190 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:40.958589 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:40.958591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:40.958591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:40.958597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:40.960195 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:40.960621 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:40.960622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:40.960622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:40.960680 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:40.960950 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:40.960951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:40.960951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:40.961062 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:40.961194 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:40.961299 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:55:40.961360 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:55:40.961879 20657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:40.972532 20657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2eb95e48a7a4dba913dae4d07ff7c87: Generated new TSK 0
I20260928 17:55:40.972725 20657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:40.976918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20577
W20260928 17:55:40.981484 20562 connection.cc:535] client connection to 127.0.0.1:36041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36041): Connection reset by peer (error 104)
2026-09-28T17:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:41 dist-test-slave-zqlt krb5kdc[20676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:41 dist-test-slave-zqlt krb5kdc[20676](info): set up 2 sockets
Sep 28 17:55:41 dist-test-slave-zqlt krb5kdc[20676](info): commencing operation
krb5kdc: starting...
W20260928 17:55:42.008380 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:42 dist-test-slave-zqlt krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:42.127802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:42.209118 20689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:42.209393 20689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:42.209434 20689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:42.210654 20689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:42.210734 20689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:42.210762 20689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:42.210804 20689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:42.210829 20689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:42.212452 20689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:42.212742 20689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:42.212989 20689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:42.217912 20694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:42.217937 20697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:42.217985 20695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:42.218026 20689 server_base.cc:1034] running on GCE node
I20260928 17:55:42.218415 20689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:42.218735 20689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:42.219903 20689 hybrid_clock.cc:648] HybridClock initialized: now 1790618142219894 us; error 27 us; skew 500 ppm
Sep 28 17:55:42 dist-test-slave-zqlt krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:42.222584 20689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:42.223069 20689 webserver.cc:460] Webserver started at http://127.0.0.1:34873/ using document root <none> and password file <none>
I20260928 17:55:42.223280 20689 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:42.223325 20689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:42.223418 20689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:42.224589 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "7b0aa334908e40be8cc56efccc0f2c09"
format_stamp: "Formatted at 2026-09-28 17:55:42 on dist-test-slave-zqlt"
server_key: "3774d89c69f2ee597c968bd5138ec02d"
server_key_iv: "942b66b4b9bf246ca3cdb920533a26fc"
server_key_version: "encryptionkey@0"
I20260928 17:55:42.224947 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b0aa334908e40be8cc56efccc0f2c09"
format_stamp: "Formatted at 2026-09-28 17:55:42 on dist-test-slave-zqlt"
server_key: "3774d89c69f2ee597c968bd5138ec02d"
server_key_iv: "942b66b4b9bf246ca3cdb920533a26fc"
server_key_version: "encryptionkey@0"
I20260928 17:55:42.226320 20689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:42.227063 20704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:42.227273 20689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:42.227397 20689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "7b0aa334908e40be8cc56efccc0f2c09"
format_stamp: "Formatted at 2026-09-28 17:55:42 on dist-test-slave-zqlt"
server_key: "3774d89c69f2ee597c968bd5138ec02d"
server_key_iv: "942b66b4b9bf246ca3cdb920533a26fc"
server_key_version: "encryptionkey@0"
I20260928 17:55:42.227494 20689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:42.243620 20689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:42.246394 20689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:42.246587 20689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:42.251017 20689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44853
I20260928 17:55:42.251053 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44853 every 8 connection(s)
I20260928 17:55:42.251443 20689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:42.252408 20757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:42.252851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20689
I20260928 17:55:42.252991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:42.253181 392 external_mini_cluster.cc:1428] Setting key 1d5ef2b643d8c47356bca1ff39a4ea07
I20260928 17:55:42.255216 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: Bootstrap starting.
I20260928 17:55:42.255857 20757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:42.256207 20757 log.cc:824] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:42.256942 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: No bootstrap required, opened a new log
I20260928 17:55:42.258818 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:42.253759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41207 (local address 127.0.0.1:44853)
0928 17:55:42.253932 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:42.253939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:42.253965 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:55:42.254357 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:42.254364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:42.254411 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:42.254528 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:42.254775 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:42.255699 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:42.256806 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:42.257251 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:42.257341 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:42.258076 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:42.258080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:42.258081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:42.258252 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:42.258294 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:42.258305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:42.258383 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:42.258511 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:55:42.258570 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260928 17:55:42.259250 20757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 17:55:42.259382 20757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:42.259433 20757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0aa334908e40be8cc56efccc0f2c09, State: Initialized, Role: FOLLOWER
I20260928 17:55:42.259563 20757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 17:55:42.259651 20757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:42.259693 20757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:42.259747 20757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:42.259999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:42.260483 20757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 17:55:42.260569 20757 leader_election.cc:302] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b0aa334908e40be8cc56efccc0f2c09; no voters:
I20260928 17:55:42.260789 20757 leader_election.cc:288] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:42.260818 20762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:42.260938 20762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [term 1 LEADER]: Becoming Leader. State: Replica: 7b0aa334908e40be8cc56efccc0f2c09, State: Running, Role: LEADER
I20260928 17:55:42.261019 20762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 17:55:42.261062 20757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:42.261322 20762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b0aa334908e40be8cc56efccc0f2c09. Latest consensus state: current_term: 1 leader_uuid: "7b0aa334908e40be8cc56efccc0f2c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } } }
I20260928 17:55:42.261392 20762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:42.261353 20764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b0aa334908e40be8cc56efccc0f2c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b0aa334908e40be8cc56efccc0f2c09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } } }
I20260928 17:55:42.261612 20764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09 [sys.catalog]: This master's current role is: LEADER
W20260928 17:55:42.262893 20782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:55:42.262943 20782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:55:42.263258 20771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:42.263376 20771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:42.264880 20771 catalog_manager.cc:1347] Generated new cluster ID: 4dba3885ecc5467f8accba419431d351
I20260928 17:55:42.264940 20771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:42.265187 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:42.261476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47069 (local address 127.0.0.1:44853)
0928 17:55:42.261510 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:42.261512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:42.261739 (+ 227us) server_negotiation.cc:407] Connection header received
0928 17:55:42.261819 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:42.261820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:42.261829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:42.261881 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:42.262015 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:42.262555 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:42.263581 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:42.263870 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:42.263943 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:42.264752 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:42.264753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:42.264754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:42.264758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:42.264773 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:42.264777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:42.264827 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:42.265105 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 17:55:42.265116 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:55:42.291579 20771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:42.292379 20771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:42.297334 20771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b0aa334908e40be8cc56efccc0f2c09: Generated new TSK 0
I20260928 17:55:42.297523 20771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:42.309420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20689
W20260928 17:55:42.313839 20674 connection.cc:535] client connection to 127.0.0.1:44853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44853): Connection reset by peer (error 104)
2026-09-28T17:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:42 dist-test-slave-zqlt krb5kdc[20788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:42 dist-test-slave-zqlt krb5kdc[20788](info): set up 2 sockets
Sep 28 17:55:42 dist-test-slave-zqlt krb5kdc[20788](info): commencing operation
krb5kdc: starting...
W20260928 17:55:43.340337 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:43 dist-test-slave-zqlt krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:43.459386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:43.541050 20801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:43.541271 20801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:43.541302 20801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:43.542482 20801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:43.542526 20801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:43.542541 20801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:43.542554 20801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:43.542616 20801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:43.544067 20801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:43.544350 20801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:43.544587 20801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:43.549540 20807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:43.549557 20806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:43.549557 20809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:43.549620 20801 server_base.cc:1034] running on GCE node
I20260928 17:55:43.550005 20801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:43.550338 20801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:43.551527 20801 hybrid_clock.cc:648] HybridClock initialized: now 1790618143551503 us; error 33 us; skew 500 ppm
Sep 28 17:55:43 dist-test-slave-zqlt krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:43.554307 20801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:43.554754 20801 webserver.cc:460] Webserver started at http://127.0.0.1:45129/ using document root <none> and password file <none>
I20260928 17:55:43.555034 20801 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:43.555109 20801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:43.555258 20801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:43.556447 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "15cbcd297b2b46989e644647f1d13db9"
format_stamp: "Formatted at 2026-09-28 17:55:43 on dist-test-slave-zqlt"
server_key: "ab39d895779099308c14e421e6a831f4"
server_key_iv: "a585a5e5892597fd291b2fb6e48373f5"
server_key_version: "encryptionkey@0"
I20260928 17:55:43.556808 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "15cbcd297b2b46989e644647f1d13db9"
format_stamp: "Formatted at 2026-09-28 17:55:43 on dist-test-slave-zqlt"
server_key: "ab39d895779099308c14e421e6a831f4"
server_key_iv: "a585a5e5892597fd291b2fb6e48373f5"
server_key_version: "encryptionkey@0"
I20260928 17:55:43.558169 20801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:43.558959 20816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:43.559187 20801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:43.559299 20801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "15cbcd297b2b46989e644647f1d13db9"
format_stamp: "Formatted at 2026-09-28 17:55:43 on dist-test-slave-zqlt"
server_key: "ab39d895779099308c14e421e6a831f4"
server_key_iv: "a585a5e5892597fd291b2fb6e48373f5"
server_key_version: "encryptionkey@0"
I20260928 17:55:43.559379 20801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:43.562835 20801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:43.564898 20801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:43.565073 20801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:43.569046 20801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41395
I20260928 17:55:43.569077 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41395 every 8 connection(s)
I20260928 17:55:43.569432 20801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:43.570464 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:43.572821 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: Bootstrap starting.
I20260928 17:55:43.573350 20869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:43.573587 20869 log.cc:824] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:43.574214 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: No bootstrap required, opened a new log
I20260928 17:55:43.574481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20801
I20260928 17:55:43.574605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:43.574813 392 external_mini_cluster.cc:1428] Setting key 8113f2bf5dbab31aa63ece0bcc821bde
I20260928 17:55:43.576424 20869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } }
I20260928 17:55:43.576592 20869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:43.576634 20869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15cbcd297b2b46989e644647f1d13db9, State: Initialized, Role: FOLLOWER
I20260928 17:55:43.576771 20869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } }
I20260928 17:55:43.576884 20869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:43.576908 20869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:43.576982 20869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:43.577829 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:43.575323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59059 (local address 127.0.0.1:41395)
0928 17:55:43.575508 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:43.575515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:43.575541 (+ 26us) server_negotiation.cc:407] Connection header received
0928 17:55:43.576408 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:43.576415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:43.576445 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:43.576566 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:43.577348 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:43.577352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:43.577352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:43.577356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:43.577389 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:43.577401 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:43.577460 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:43.577560 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:55:43.577620 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 17:55:43.578126 20869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } }
I20260928 17:55:43.578258 20869 leader_election.cc:302] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15cbcd297b2b46989e644647f1d13db9; no voters:
I20260928 17:55:43.578440 20869 leader_election.cc:288] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:43.578480 20874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:43.578619 20874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [term 1 LEADER]: Becoming Leader. State: Replica: 15cbcd297b2b46989e644647f1d13db9, State: Running, Role: LEADER
I20260928 17:55:43.578716 20869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:43.578708 20874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } }
I20260928 17:55:43.578774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:43.579015 20874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15cbcd297b2b46989e644647f1d13db9. Latest consensus state: current_term: 1 leader_uuid: "15cbcd297b2b46989e644647f1d13db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } } }
I20260928 17:55:43.579025 20875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15cbcd297b2b46989e644647f1d13db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cbcd297b2b46989e644647f1d13db9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41395 } } }
I20260928 17:55:43.579101 20874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:43.579103 20875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9 [sys.catalog]: This master's current role is: LEADER
W20260928 17:55:43.580627 20893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:55:43.580693 20893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:55:43.580813 20894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:43.580971 20894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:43.582528 20894 catalog_manager.cc:1347] Generated new cluster ID: 68da67fe908643bda52df373e9af89ad
I20260928 17:55:43.582634 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:43.580456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47833 (local address 127.0.0.1:41395)
0928 17:55:43.580486 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:43.580488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:43.581604 (+ 1116us) server_negotiation.cc:407] Connection header received
0928 17:55:43.581635 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:43.581636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:43.581643 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:43.581705 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:43.582352 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:43.582354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:43.582354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:43.582358 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:43.582372 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:43.582377 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:43.582408 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:43.582533 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 17:55:43.582547 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 17:55:43.582589 20894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:43.587265 20894 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:43.587466 20894 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:43.592211 20894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15cbcd297b2b46989e644647f1d13db9: Generated new TSK 0
I20260928 17:55:43.592381 20894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:43.596158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20801
2026-09-28T17:55:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:43 dist-test-slave-zqlt krb5kdc[20900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:43 dist-test-slave-zqlt krb5kdc[20900](info): set up 2 sockets
Sep 28 17:55:43 dist-test-slave-zqlt krb5kdc[20900](info): commencing operation
krb5kdc: starting...
W20260928 17:55:44.628304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:44.745257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:44.826925 20913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:44.827167 20913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:44.827190 20913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:44.828431 20913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:44.828487 20913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:44.828502 20913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:44.828516 20913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:44.828528 20913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:44.830066 20913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:44.830344 20913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:44.830597 20913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:44.835726 20918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:44.835745 20921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:44.835789 20919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:44.836249 20913 server_base.cc:1034] running on GCE node
I20260928 17:55:44.836472 20913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:44.836730 20913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:44.837872 20913 hybrid_clock.cc:648] HybridClock initialized: now 1790618144837861 us; error 29 us; skew 500 ppm
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:44.843940 20913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:44.844532 20913 webserver.cc:460] Webserver started at http://127.0.0.1:42343/ using document root <none> and password file <none>
I20260928 17:55:44.844769 20913 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:44.844841 20913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:44.845003 20913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:44.846120 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "25f7a3cbb72f42e5a3180f4d4472a28e"
format_stamp: "Formatted at 2026-09-28 17:55:44 on dist-test-slave-zqlt"
server_key: "85e12fe49f181b3c65f204d81566aec5"
server_key_iv: "2711f66d0b790540e3b6b62ee7523fe2"
server_key_version: "encryptionkey@0"
I20260928 17:55:44.846530 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "25f7a3cbb72f42e5a3180f4d4472a28e"
format_stamp: "Formatted at 2026-09-28 17:55:44 on dist-test-slave-zqlt"
server_key: "85e12fe49f181b3c65f204d81566aec5"
server_key_iv: "2711f66d0b790540e3b6b62ee7523fe2"
server_key_version: "encryptionkey@0"
I20260928 17:55:44.847904 20913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:44.848723 20928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:44.848899 20913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:44.849015 20913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "25f7a3cbb72f42e5a3180f4d4472a28e"
format_stamp: "Formatted at 2026-09-28 17:55:44 on dist-test-slave-zqlt"
server_key: "85e12fe49f181b3c65f204d81566aec5"
server_key_iv: "2711f66d0b790540e3b6b62ee7523fe2"
server_key_version: "encryptionkey@0"
I20260928 17:55:44.849088 20913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:44.865898 20913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:44.867967 20913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:44.868146 20913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:44.872046 20913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38077
I20260928 17:55:44.872149 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38077 every 8 connection(s)
I20260928 17:55:44.872511 20913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:44.873394 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:44.875608 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e: Bootstrap starting.
I20260928 17:55:44.876219 20981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:44.876462 20981 log.cc:824] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:44.877125 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e: No bootstrap required, opened a new log
I20260928 17:55:44.878644 20981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 17:55:44.878778 20981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:44.878800 20981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f7a3cbb72f42e5a3180f4d4472a28e, State: Initialized, Role: FOLLOWER
I20260928 17:55:44.878949 20981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 17:55:44.879034 20981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:44.879057 20981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:44.879086 20981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:44.879560 20981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 17:55:44.879643 20981 leader_election.cc:302] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f7a3cbb72f42e5a3180f4d4472a28e; no voters:
I20260928 17:55:44.879793 20981 leader_election.cc:288] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:44.879829 20984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:44.879971 20984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [term 1 LEADER]: Becoming Leader. State: Replica: 25f7a3cbb72f42e5a3180f4d4472a28e, State: Running, Role: LEADER
I20260928 17:55:44.880051 20981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:44.880136 20984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } }
I20260928 17:55:44.880273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 20913
I20260928 17:55:44.880417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:44.880599 392 external_mini_cluster.cc:1428] Setting key afcb05ceb53231164fd82ef23f4c84ef
I20260928 17:55:44.880810 20985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260928 17:55:44.880963 20985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:44.881237 20995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:44.881506 20986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f7a3cbb72f42e5a3180f4d4472a28e. Latest consensus state: current_term: 1 leader_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f7a3cbb72f42e5a3180f4d4472a28e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38077 } } }
I20260928 17:55:44.881632 20986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:44.881806 20995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:44.884496 20995 catalog_manager.cc:1347] Generated new cluster ID: 7dc0989474e748d4a37361f47484d276
I20260928 17:55:44.884548 20995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[20900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:55:44.892423 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:44.881273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52799 (local address 127.0.0.1:38077)
0928 17:55:44.881454 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:44.881463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:44.881891 (+ 428us) server_negotiation.cc:407] Connection header received
0928 17:55:44.882051 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:44.882059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:44.882115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:44.882253 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:44.883109 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:44.884029 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:44.885123 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:44.885754 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:44.885854 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:44.888693 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:44.888697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:44.888698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:44.888709 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:44.890778 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:44.891146 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:44.891149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:44.891149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:44.891267 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:44.891472 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:44.891473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:44.891473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:44.891792 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:44.891963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:44.892039 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 17:55:44.892204 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260928 17:55:44.893379 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:44.896198 20995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:44.896644 20995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:44.901234 20995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f7a3cbb72f42e5a3180f4d4472a28e: Generated new TSK 0
I20260928 17:55:44.901476 20995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:44.902024 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:44.894816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44577 (local address 127.0.0.1:38077)
0928 17:55:44.894883 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:44.894885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:44.895073 (+ 188us) server_negotiation.cc:407] Connection header received
0928 17:55:44.895119 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:44.895120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:44.895130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:44.895186 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:44.895794 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:44.896557 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:44.897589 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:44.897900 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:44.897966 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:44.899301 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:44.899302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:44.899302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:44.899309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:44.900803 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:44.901254 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:44.901255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:44.901256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:44.901320 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:44.901542 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:44.901543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:44.901543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:44.901666 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:44.901765 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:44.901893 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 17:55:44.901969 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 17:55:44.903798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 20913
2026-09-28T17:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[21012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[21012](info): set up 2 sockets
Sep 28 17:55:44 dist-test-slave-zqlt krb5kdc[21012](info): commencing operation
krb5kdc: starting...
W20260928 17:55:45.932125 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:46 dist-test-slave-zqlt krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:46.055258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:46.137036 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:46.137247 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:46.137270 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:46.138495 21025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:46.138540 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:46.138556 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:46.138568 21025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:46.138581 21025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:46.140067 21025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:46.140357 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:46.140590 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:46.145424 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:46.145464 21025 server_base.cc:1034] running on GCE node
W20260928 17:55:46.145454 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:46.145437 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:46.145836 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:46.146098 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:46.147248 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790618146147225 us; error 39 us; skew 500 ppm
Sep 28 17:55:46 dist-test-slave-zqlt krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:46.149729 21025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:46.150117 21025 webserver.cc:460] Webserver started at http://127.0.0.1:37913/ using document root <none> and password file <none>
I20260928 17:55:46.150312 21025 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:46.150352 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:46.150439 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:46.151407 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "1fb5a1e499b24f3db732fff31cf5518d"
format_stamp: "Formatted at 2026-09-28 17:55:46 on dist-test-slave-zqlt"
server_key: "3728e4ac21b18cac27663f6d4fe6c2f7"
server_key_iv: "ccca2f2b4a30f9498854e6b8406d2fde"
server_key_version: "encryptionkey@0"
I20260928 17:55:46.151715 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fb5a1e499b24f3db732fff31cf5518d"
format_stamp: "Formatted at 2026-09-28 17:55:46 on dist-test-slave-zqlt"
server_key: "3728e4ac21b18cac27663f6d4fe6c2f7"
server_key_iv: "ccca2f2b4a30f9498854e6b8406d2fde"
server_key_version: "encryptionkey@0"
I20260928 17:55:46.153083 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:46.153944 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:46.154122 21025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:46.154240 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "1fb5a1e499b24f3db732fff31cf5518d"
format_stamp: "Formatted at 2026-09-28 17:55:46 on dist-test-slave-zqlt"
server_key: "3728e4ac21b18cac27663f6d4fe6c2f7"
server_key_iv: "ccca2f2b4a30f9498854e6b8406d2fde"
server_key_version: "encryptionkey@0"
I20260928 17:55:46.154327 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:46.166530 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:46.168547 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:46.168710 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:46.172959 21025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34353
I20260928 17:55:46.172988 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34353 every 8 connection(s)
I20260928 17:55:46.173357 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:46.174253 21093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:46.177028 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d: Bootstrap starting.
I20260928 17:55:46.177604 21093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:46.177891 21093 log.cc:824] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:46.178566 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d: No bootstrap required, opened a new log
I20260928 17:55:46.180035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21025
I20260928 17:55:46.180199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:46.180078 21093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260928 17:55:46.180298 21093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:46.180346 21093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb5a1e499b24f3db732fff31cf5518d, State: Initialized, Role: FOLLOWER
I20260928 17:55:46.180429 392 external_mini_cluster.cc:1428] Setting key 1d02ce860b9ba6860d4c154765cce8dd
I20260928 17:55:46.180531 21093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260928 17:55:46.180632 21093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:46.180691 21093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:46.180745 21093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:46.181643 21093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260928 17:55:46.181759 21093 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb5a1e499b24f3db732fff31cf5518d; no voters:
I20260928 17:55:46.181923 21093 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:46.181979 21098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:46.182113 21098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [term 1 LEADER]: Becoming Leader. State: Replica: 1fb5a1e499b24f3db732fff31cf5518d, State: Running, Role: LEADER
I20260928 17:55:46.182224 21093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:46.182235 21098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } }
I20260928 17:55:46.182680 21098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb5a1e499b24f3db732fff31cf5518d. Latest consensus state: current_term: 1 leader_uuid: "1fb5a1e499b24f3db732fff31cf5518d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } } }
I20260928 17:55:46.182689 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb5a1e499b24f3db732fff31cf5518d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb5a1e499b24f3db732fff31cf5518d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34353 } } }
I20260928 17:55:46.182858 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:46.183054 21098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:46.183200 21107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:46.183771 21107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:46.185376 21107 catalog_manager.cc:1347] Generated new cluster ID: 21c18968f8ac4cc28d311cdef0eb62c9
I20260928 17:55:46.185429 21107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:46.186344 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:46.180947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56299 (local address 127.0.0.1:34353)
0928 17:55:46.181123 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:46.181133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:46.181167 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:55:46.181285 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:46.181290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:46.181331 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:46.181471 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:46.181678 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:46.182976 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:46.184130 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:46.184751 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:46.184861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:46.185699 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:46.185703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:46.185703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:46.185900 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:46.185936 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:46.185948 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:46.186026 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:46.186109 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 17:55:46.186173 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 17:55:46.187348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:46.190505 21107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:46.191090 21107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:46.192239 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:46.188781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55017 (local address 127.0.0.1:34353)
0928 17:55:46.188836 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:46.188838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:46.188965 (+ 127us) server_negotiation.cc:407] Connection header received
0928 17:55:46.189009 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:46.189010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:46.189021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:46.189090 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:46.189197 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:46.189950 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:46.190875 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:46.191187 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:46.191276 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:46.191956 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:46.191958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:46.191958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:46.191962 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:46.191977 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:46.191981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:46.192042 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:46.192126 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 17:55:46.192170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:55:46.196867 21107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb5a1e499b24f3db732fff31cf5518d: Generated new TSK 0
I20260928 17:55:46.197041 21107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:46.200305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21025
2026-09-28T17:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:46 dist-test-slave-zqlt krb5kdc[21124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:46 dist-test-slave-zqlt krb5kdc[21124](info): set up 2 sockets
Sep 28 17:55:46 dist-test-slave-zqlt krb5kdc[21124](info): commencing operation
krb5kdc: starting...
W20260928 17:55:47.228284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:47 dist-test-slave-zqlt krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:47.346812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:47.427387 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:47.427618 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:47.427641 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:47.428874 21137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:47.428921 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:47.428936 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:47.428947 21137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:47.428967 21137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:47.430365 21137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:47.430570 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:47.430743 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:47.435726 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:47.435753 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:47.435727 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:47.435726 21137 server_base.cc:1034] running on GCE node
I20260928 17:55:47.436172 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:47.436455 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:47.437603 21137 hybrid_clock.cc:648] HybridClock initialized: now 1790618147437594 us; error 30 us; skew 500 ppm
Sep 28 17:55:47 dist-test-slave-zqlt krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:47.440164 21137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:47.440647 21137 webserver.cc:460] Webserver started at http://127.0.0.1:40495/ using document root <none> and password file <none>
I20260928 17:55:47.440876 21137 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:47.440922 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:47.441069 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:47.442160 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "4886ffd488ed405eb0e74395ef58e3ad"
format_stamp: "Formatted at 2026-09-28 17:55:47 on dist-test-slave-zqlt"
server_key: "f819ee7495bb13971b2f2bd274ce10d1"
server_key_iv: "d5fc7b12b61b5f6fd6c61526e51f2d9e"
server_key_version: "encryptionkey@0"
I20260928 17:55:47.442538 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "4886ffd488ed405eb0e74395ef58e3ad"
format_stamp: "Formatted at 2026-09-28 17:55:47 on dist-test-slave-zqlt"
server_key: "f819ee7495bb13971b2f2bd274ce10d1"
server_key_iv: "d5fc7b12b61b5f6fd6c61526e51f2d9e"
server_key_version: "encryptionkey@0"
I20260928 17:55:47.443912 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:55:47.444741 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:47.444934 21137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:47.445040 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "4886ffd488ed405eb0e74395ef58e3ad"
format_stamp: "Formatted at 2026-09-28 17:55:47 on dist-test-slave-zqlt"
server_key: "f819ee7495bb13971b2f2bd274ce10d1"
server_key_iv: "d5fc7b12b61b5f6fd6c61526e51f2d9e"
server_key_version: "encryptionkey@0"
I20260928 17:55:47.445115 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:47.448470 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:47.450412 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:47.450558 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:47.454702 21137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43741
I20260928 17:55:47.454741 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43741 every 8 connection(s)
I20260928 17:55:47.455106 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:47.456218 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:47.458547 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad: Bootstrap starting.
I20260928 17:55:47.459136 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:47.459425 21205 log.cc:824] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:47.460350 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad: No bootstrap required, opened a new log
I20260928 17:55:47.461521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21137
I20260928 17:55:47.461665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:47.461874 392 external_mini_cluster.cc:1428] Setting key d233c45ebf9139bd310501f85ee43afb
I20260928 17:55:47.462350 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260928 17:55:47.462607 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:47.462674 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4886ffd488ed405eb0e74395ef58e3ad, State: Initialized, Role: FOLLOWER
I20260928 17:55:47.462805 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260928 17:55:47.462896 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:47.462930 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:47.462977 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:47.464143 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260928 17:55:47.464247 21205 leader_election.cc:302] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4886ffd488ed405eb0e74395ef58e3ad; no voters:
I20260928 17:55:47.464378 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:47.462568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46065 (local address 127.0.0.1:43741)
0928 17:55:47.462845 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:47.462855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:47.462890 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:55:47.463025 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:47.463033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:47.463065 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:47.463228 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:47.463898 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:47.463903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:47.463903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:47.463908 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:47.463952 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:47.463967 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:47.464049 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:47.464114 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 17:55:47.464160 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260928 17:55:47.464427 21205 leader_election.cc:288] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:47.464490 21210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:47.464655 21210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [term 1 LEADER]: Becoming Leader. State: Replica: 4886ffd488ed405eb0e74395ef58e3ad, State: Running, Role: LEADER
I20260928 17:55:47.464668 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:47.464798 21210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260928 17:55:47.465224 21211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4886ffd488ed405eb0e74395ef58e3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } } }
I20260928 17:55:47.465224 21212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4886ffd488ed405eb0e74395ef58e3ad. Latest consensus state: current_term: 1 leader_uuid: "4886ffd488ed405eb0e74395ef58e3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886ffd488ed405eb0e74395ef58e3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } } }
I20260928 17:55:47.465333 21211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:47.465375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:47.465339 21212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:47.466394 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:47.467018 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:47.468425 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:47.466902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47325 (local address 127.0.0.1:43741)
0928 17:55:47.466936 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:47.466938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:47.467237 (+ 299us) server_negotiation.cc:407] Connection header received
0928 17:55:47.467298 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:47.467299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:47.467304 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:47.467353 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:47.468190 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:47.468191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:47.468192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:47.468196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:47.468209 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:47.468213 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:47.468286 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:47.468349 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 17:55:47.468363 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:55:47.468703 21218 catalog_manager.cc:1347] Generated new cluster ID: 12a693b24526402ab94625570e06657c
I20260928 17:55:47.468750 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:47.476137 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:47.476320 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:47.481395 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4886ffd488ed405eb0e74395ef58e3ad: Generated new TSK 0
I20260928 17:55:47.481592 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:47.486032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21137
2026-09-28T17:55:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:47 dist-test-slave-zqlt krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:47 dist-test-slave-zqlt krb5kdc[21236](info): set up 2 sockets
Sep 28 17:55:47 dist-test-slave-zqlt krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260928 17:55:48.516312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:48.649127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:48.731065 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:48.731316 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:48.731362 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:48.732574 21249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:48.732641 21249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:48.732664 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:48.732683 21249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:48.732715 21249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:48.734180 21249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:48.734484 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:48.734704 21249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:48.739652 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:48.739652 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:48.739640 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:48.739992 21249 server_base.cc:1034] running on GCE node
I20260928 17:55:48.740260 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:48.740576 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:48.741729 21249 hybrid_clock.cc:648] HybridClock initialized: now 1790618148741706 us; error 41 us; skew 500 ppm
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:48.744256 21249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:48.744730 21249 webserver.cc:460] Webserver started at http://127.0.0.1:33985/ using document root <none> and password file <none>
I20260928 17:55:48.744958 21249 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:48.745033 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:48.745168 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:48.746230 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "a20e2252e1554cd0aba96aaa8db384e9"
format_stamp: "Formatted at 2026-09-28 17:55:48 on dist-test-slave-zqlt"
server_key: "302ab59f5cdd4664b21bb029c0a505e0"
server_key_iv: "fa83a6de8d9e1e4261ab763b965dd986"
server_key_version: "encryptionkey@0"
I20260928 17:55:48.746603 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "a20e2252e1554cd0aba96aaa8db384e9"
format_stamp: "Formatted at 2026-09-28 17:55:48 on dist-test-slave-zqlt"
server_key: "302ab59f5cdd4664b21bb029c0a505e0"
server_key_iv: "fa83a6de8d9e1e4261ab763b965dd986"
server_key_version: "encryptionkey@0"
I20260928 17:55:48.747896 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:55:48.748728 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:48.748911 21249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:48.749024 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "a20e2252e1554cd0aba96aaa8db384e9"
format_stamp: "Formatted at 2026-09-28 17:55:48 on dist-test-slave-zqlt"
server_key: "302ab59f5cdd4664b21bb029c0a505e0"
server_key_iv: "fa83a6de8d9e1e4261ab763b965dd986"
server_key_version: "encryptionkey@0"
I20260928 17:55:48.749109 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:48.766563 21249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:48.768615 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:48.768774 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:48.772792 21249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42843
I20260928 17:55:48.772827 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42843 every 8 connection(s)
I20260928 17:55:48.773362 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:48.774017 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:48.774413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21249
I20260928 17:55:48.774540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:48.774742 392 external_mini_cluster.cc:1428] Setting key 1a009fb576f76c4e98319a03ea8f2fca
I20260928 17:55:48.777215 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9: Bootstrap starting.
I20260928 17:55:48.777808 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:48.778103 21317 log.cc:824] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:48.778769 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9: No bootstrap required, opened a new log
I20260928 17:55:48.780618 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260928 17:55:48.780771 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:48.780818 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e2252e1554cd0aba96aaa8db384e9, State: Initialized, Role: FOLLOWER
I20260928 17:55:48.781016 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260928 17:55:48.781116 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 17:55:48.781203 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:55:48.781311 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:48.781358 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:48.781316 21322 leader_election.cc:302] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20e2252e1554cd0aba96aaa8db384e9; no voters:
I20260928 17:55:48.781397 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:48.781525 21322 leader_election.cc:288] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 17:55:48.782019 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260928 17:55:48.782107 21317 leader_election.cc:302] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20e2252e1554cd0aba96aaa8db384e9; no voters:
I20260928 17:55:48.782158 21317 leader_election.cc:288] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:48.782173 21322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:48.782277 21322 raft_consensus.cc:686] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 1 LEADER]: Becoming Leader. State: Replica: a20e2252e1554cd0aba96aaa8db384e9, State: Running, Role: LEADER
I20260928 17:55:48.782361 21322 consensus_queue.cc:237] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } }
I20260928 17:55:48.782415 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:48.782765 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20e2252e1554cd0aba96aaa8db384e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } } }
I20260928 17:55:48.782886 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20e2252e1554cd0aba96aaa8db384e9. Latest consensus state: current_term: 1 leader_uuid: "a20e2252e1554cd0aba96aaa8db384e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20e2252e1554cd0aba96aaa8db384e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42843 } } }
I20260928 17:55:48.782951 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:48.782915 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:48.783178 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:48.783670 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:48.783811 21323 raft_consensus.cc:2741] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9 [term 1 LEADER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 17:55:48.785590 21330 catalog_manager.cc:1347] Generated new cluster ID: 44104436fb0d46f79c0ab9a74d520b5f
I20260928 17:55:48.785837 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:48.775279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:58441 (local address 127.0.0.1:42843)
0928 17:55:48.775449 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:48.775457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:48.775484 (+ 27us) server_negotiation.cc:407] Connection header received
0928 17:55:48.775881 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:48.775886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:48.775934 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:48.776064 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:48.776917 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:48.777835 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:48.778660 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:48.779143 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:48.779263 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:48.781866 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:48.781869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:48.781869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:48.781879 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:48.783995 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:48.784541 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:48.784544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:48.784544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:48.784642 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:48.784881 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:48.784882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:48.784882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:48.785218 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:48.785411 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:48.785509 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 17:55:48.785631 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 17:55:48.785653 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:48.786898 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:48.799757 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:48.788488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:57621 (local address 127.0.0.1:42843)
0928 17:55:48.792177 (+ 3689us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:48.792179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:48.792187 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:48.792213 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:48.792215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:48.792226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:48.792284 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:48.792999 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:48.793756 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:48.794836 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:48.795163 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:48.795226 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:48.796793 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:48.796795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:48.796795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:48.796801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:48.798410 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:48.798838 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:48.798839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:48.798839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:48.798906 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:48.799139 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:48.799140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:48.799140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:48.799258 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:48.799353 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:48.799506 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 17:55:48.799565 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3664}
I20260928 17:55:48.816934 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:48.817529 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:48.831051 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a20e2252e1554cd0aba96aaa8db384e9: Generated new TSK 0
I20260928 17:55:48.831315 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:48.844255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21249
W20260928 17:55:48.849211 21234 connection.cc:535] client connection to 127.0.0.1:42843 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42843): Connection reset by peer (error 104)
2026-09-28T17:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21349](info): set up 2 sockets
Sep 28 17:55:48 dist-test-slave-zqlt krb5kdc[21349](info): commencing operation
krb5kdc: starting...
W20260928 17:55:49.876312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:49 dist-test-slave-zqlt krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:49.994794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:50.075346 21362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:50.075611 21362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:50.075652 21362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:50.076880 21362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:50.076953 21362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:50.076985 21362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:50.077013 21362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:50.077037 21362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:50.078490 21362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:50.078763 21362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:50.079006 21362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:50.083739 21367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:50.083729 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:50.083730 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:50.083773 21362 server_base.cc:1034] running on GCE node
I20260928 17:55:50.084182 21362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:50.084466 21362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:50.085635 21362 hybrid_clock.cc:648] HybridClock initialized: now 1790618150085626 us; error 40 us; skew 500 ppm
Sep 28 17:55:50 dist-test-slave-zqlt krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:50.092743 21362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:50.093211 21362 webserver.cc:460] Webserver started at http://127.0.0.1:45073/ using document root <none> and password file <none>
I20260928 17:55:50.093407 21362 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:50.093447 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:50.093533 21362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:50.094554 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "1b69376fd22d4868b11d629d322cdbfa"
format_stamp: "Formatted at 2026-09-28 17:55:50 on dist-test-slave-zqlt"
server_key: "ae66d7029215e8bc4441918b1f930998"
server_key_iv: "da163f27b3a77b10be00e5d5554fb448"
server_key_version: "encryptionkey@0"
I20260928 17:55:50.094859 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b69376fd22d4868b11d629d322cdbfa"
format_stamp: "Formatted at 2026-09-28 17:55:50 on dist-test-slave-zqlt"
server_key: "ae66d7029215e8bc4441918b1f930998"
server_key_iv: "da163f27b3a77b10be00e5d5554fb448"
server_key_version: "encryptionkey@0"
I20260928 17:55:50.096179 21362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:50.096951 21377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:50.097163 21362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 17:55:50.097259 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "1b69376fd22d4868b11d629d322cdbfa"
format_stamp: "Formatted at 2026-09-28 17:55:50 on dist-test-slave-zqlt"
server_key: "ae66d7029215e8bc4441918b1f930998"
server_key_iv: "da163f27b3a77b10be00e5d5554fb448"
server_key_version: "encryptionkey@0"
I20260928 17:55:50.097316 21362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:50.126400 21362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:50.129673 21362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:50.129835 21362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:50.134095 21362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36345
I20260928 17:55:50.134135 21429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36345 every 8 connection(s)
I20260928 17:55:50.134529 21362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:50.135449 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:50.137858 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa: Bootstrap starting.
I20260928 17:55:50.138466 21430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:50.138758 21430 log.cc:824] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:50.139487 21430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa: No bootstrap required, opened a new log
I20260928 17:55:50.139742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21362
I20260928 17:55:50.139900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:50.140146 392 external_mini_cluster.cc:1428] Setting key 844cfd28b83fc2966e6bbba135b923b2
I20260928 17:55:50.142040 21430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } }
I20260928 17:55:50.142266 21430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:50.142300 21430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69376fd22d4868b11d629d322cdbfa, State: Initialized, Role: FOLLOWER
I20260928 17:55:50.142428 21430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } }
I20260928 17:55:50.142515 21430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:50.142549 21430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:50.142637 21430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:50.143397 21430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } }
I20260928 17:55:50.143510 21430 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69376fd22d4868b11d629d322cdbfa; no voters:
I20260928 17:55:50.143683 21430 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:50.143774 21435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:50.143949 21435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [term 1 LEADER]: Becoming Leader. State: Replica: 1b69376fd22d4868b11d629d322cdbfa, State: Running, Role: LEADER
I20260928 17:55:50.143998 21430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:50.144124 21435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } }
I20260928 17:55:50.144541 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b69376fd22d4868b11d629d322cdbfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } } }
I20260928 17:55:50.144524 21435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69376fd22d4868b11d629d322cdbfa. Latest consensus state: current_term: 1 leader_uuid: "1b69376fd22d4868b11d629d322cdbfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69376fd22d4868b11d629d322cdbfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36345 } } }
I20260928 17:55:50.144618 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:50.144624 21435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:50.144866 21443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:50.145412 21443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:50.147117 21443 catalog_manager.cc:1347] Generated new cluster ID: 0ce4772ae75f4b41ad68e023ee418b05
I20260928 17:55:50.147168 21443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:50.147593 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:50.140699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:38655 (local address 127.0.0.1:36345)
0928 17:55:50.140925 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:50.140936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:50.140971 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:55:50.142239 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:50.142250 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:50.142323 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:50.142491 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:50.142734 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:50.144115 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:50.145359 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:50.145936 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:50.146071 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:50.146822 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:50.146826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:50.146826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:50.147058 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:50.147097 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:50.147112 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:50.147199 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:50.147361 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:55:50.147389 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":127,"threads_started":1}
I20260928 17:55:50.148655 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:50.151881 21443 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:50.152452 21443 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:50.153775 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:50.150190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:58411 (local address 127.0.0.1:36345)
0928 17:55:50.150228 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:50.150230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:50.150425 (+ 195us) server_negotiation.cc:407] Connection header received
0928 17:55:50.150471 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:50.150472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:50.150491 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:50.150548 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:50.150682 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:50.151359 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:50.152501 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:50.152760 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:50.152823 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:50.153515 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:50.153516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:50.153517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:50.153521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:50.153534 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:50.153538 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:50.153590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:50.153697 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 17:55:50.153706 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:55:50.161420 21443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b69376fd22d4868b11d629d322cdbfa: Generated new TSK 0
I20260928 17:55:50.161603 21443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:50.166152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21362
W20260928 17:55:50.170502 21347 connection.cc:535] client connection to 127.0.0.1:36345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36345): Connection reset by peer (error 104)
2026-09-28T17:55:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:50 dist-test-slave-zqlt krb5kdc[21461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:50 dist-test-slave-zqlt krb5kdc[21461](info): set up 2 sockets
Sep 28 17:55:50 dist-test-slave-zqlt krb5kdc[21461](info): commencing operation
krb5kdc: starting...
W20260928 17:55:51.224445 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:51 dist-test-slave-zqlt krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:51.345146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:51.426496 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:51.426700 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:51.426723 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:51.427767 21474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:51.427800 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:51.427814 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:51.427826 21474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:51.427837 21474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:51.429179 21474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:51.429426 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:51.429598 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:51.434465 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:51.434425 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:51.434476 21474 server_base.cc:1034] running on GCE node
W20260928 17:55:51.434445 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:51.434950 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:51.435266 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:51.436403 21474 hybrid_clock.cc:648] HybridClock initialized: now 1790618151436398 us; error 34 us; skew 500 ppm
Sep 28 17:55:51 dist-test-slave-zqlt krb5kdc[21461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:51.438848 21474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:51.439342 21474 webserver.cc:460] Webserver started at http://127.0.0.1:36417/ using document root <none> and password file <none>
I20260928 17:55:51.439612 21474 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:51.439682 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:51.439831 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:51.440994 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "5333c988060d486e990478ffbd2b55ee"
format_stamp: "Formatted at 2026-09-28 17:55:51 on dist-test-slave-zqlt"
server_key: "246c35c8e6d0ea30e56fd3170d410268"
server_key_iv: "831d5ac83695e14339122c6bd779a439"
server_key_version: "encryptionkey@0"
I20260928 17:55:51.441354 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "5333c988060d486e990478ffbd2b55ee"
format_stamp: "Formatted at 2026-09-28 17:55:51 on dist-test-slave-zqlt"
server_key: "246c35c8e6d0ea30e56fd3170d410268"
server_key_iv: "831d5ac83695e14339122c6bd779a439"
server_key_version: "encryptionkey@0"
I20260928 17:55:51.442701 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:55:51.443454 21489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:51.443625 21474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:55:51.443723 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "5333c988060d486e990478ffbd2b55ee"
format_stamp: "Formatted at 2026-09-28 17:55:51 on dist-test-slave-zqlt"
server_key: "246c35c8e6d0ea30e56fd3170d410268"
server_key_iv: "831d5ac83695e14339122c6bd779a439"
server_key_version: "encryptionkey@0"
I20260928 17:55:51.443806 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:51.447185 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:51.449446 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:51.449672 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:51.454370 21474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42295
I20260928 17:55:51.454394 21541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42295 every 8 connection(s)
I20260928 17:55:51.454833 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:51.455880 21542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:51.458211 21542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee: Bootstrap starting.
I20260928 17:55:51.458814 21542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:51.459110 21542 log.cc:824] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:51.459848 21542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee: No bootstrap required, opened a new log
I20260928 17:55:51.459928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21474
I20260928 17:55:51.460053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:51.460268 392 external_mini_cluster.cc:1428] Setting key 0e461fe2ccfac01acf45f93d276b2842
I20260928 17:55:51.461882 21542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } }
I20260928 17:55:51.462072 21542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:51.462121 21542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5333c988060d486e990478ffbd2b55ee, State: Initialized, Role: FOLLOWER
I20260928 17:55:51.462303 21542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } }
I20260928 17:55:51.462401 21542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:51.462425 21542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:51.462495 21542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:51.463276 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:51.460835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:49983 (local address 127.0.0.1:42295)
0928 17:55:51.461018 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:51.461027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:51.461066 (+ 39us) server_negotiation.cc:407] Connection header received
0928 17:55:51.461865 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:51.461871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:51.461897 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:51.462016 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:51.462763 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:51.462768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:51.462768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:51.462772 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:51.462817 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:51.462831 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:51.462919 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:51.462961 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 17:55:51.463010 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260928 17:55:51.463685 21542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } }
I20260928 17:55:51.463788 21542 leader_election.cc:302] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5333c988060d486e990478ffbd2b55ee; no voters:
I20260928 17:55:51.463938 21542 leader_election.cc:288] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:51.464004 21547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:51.464175 21547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [term 1 LEADER]: Becoming Leader. State: Replica: 5333c988060d486e990478ffbd2b55ee, State: Running, Role: LEADER
I20260928 17:55:51.464248 21542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:51.464296 21547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } }
I20260928 17:55:51.464326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:51.464594 21547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5333c988060d486e990478ffbd2b55ee. Latest consensus state: current_term: 1 leader_uuid: "5333c988060d486e990478ffbd2b55ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } } }
I20260928 17:55:51.464649 21547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:51.464851 21552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:51.465531 21552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:51.466009 21548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5333c988060d486e990478ffbd2b55ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5333c988060d486e990478ffbd2b55ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42295 } } }
I20260928 17:55:51.466089 21548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:51.467446 21552 catalog_manager.cc:1347] Generated new cluster ID: 385bf507c4ba459f97df7de3f301a398
I20260928 17:55:51.467509 21552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:51.467566 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:51.466304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:38935 (local address 127.0.0.1:42295)
0928 17:55:51.466340 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:51.466342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:51.466455 (+ 113us) server_negotiation.cc:407] Connection header received
0928 17:55:51.466540 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:51.466540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:51.466546 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:51.466586 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:51.467199 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:51.467200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:51.467201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:51.467205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:51.467221 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:51.467225 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:51.467282 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:51.467437 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 17:55:51.467451 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:55:51.486743 21552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:51.486936 21552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:51.499485 21552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5333c988060d486e990478ffbd2b55ee: Generated new TSK 0
I20260928 17:55:51.499688 21552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:51.503180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21474
2026-09-28T17:55:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:51 dist-test-slave-zqlt krb5kdc[21573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:51 dist-test-slave-zqlt krb5kdc[21573](info): set up 2 sockets
Sep 28 17:55:51 dist-test-slave-zqlt krb5kdc[21573](info): commencing operation
krb5kdc: starting...
W20260928 17:55:52.524513 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:52.645334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:52.726703 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:52.726912 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:52.726934 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:52.728113 21586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:52.728157 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:52.728173 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:52.728186 21586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:52.728199 21586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:52.729745 21586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:52.729990 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:52.730221 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:52.735134 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:52.735139 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:52.735153 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:52.735433 21586 server_base.cc:1034] running on GCE node
I20260928 17:55:52.735752 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:52.736140 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:52.737298 21586 hybrid_clock.cc:648] HybridClock initialized: now 1790618152737287 us; error 32 us; skew 500 ppm
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:52.739751 21586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:52.740310 21586 webserver.cc:460] Webserver started at http://127.0.0.1:33511/ using document root <none> and password file <none>
I20260928 17:55:52.740567 21586 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:52.740638 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:52.740789 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:52.741868 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "82c5813af2d44f539eed8931c6867c4b"
format_stamp: "Formatted at 2026-09-28 17:55:52 on dist-test-slave-zqlt"
server_key: "77b04a049de9dcc5c8166cf4876be7a0"
server_key_iv: "a06ea4e2509c29e3a3476a81d5c2c57c"
server_key_version: "encryptionkey@0"
I20260928 17:55:52.742239 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "82c5813af2d44f539eed8931c6867c4b"
format_stamp: "Formatted at 2026-09-28 17:55:52 on dist-test-slave-zqlt"
server_key: "77b04a049de9dcc5c8166cf4876be7a0"
server_key_iv: "a06ea4e2509c29e3a3476a81d5c2c57c"
server_key_version: "encryptionkey@0"
I20260928 17:55:52.743620 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 17:55:52.744421 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:52.744611 21586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:52.744720 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "82c5813af2d44f539eed8931c6867c4b"
format_stamp: "Formatted at 2026-09-28 17:55:52 on dist-test-slave-zqlt"
server_key: "77b04a049de9dcc5c8166cf4876be7a0"
server_key_iv: "a06ea4e2509c29e3a3476a81d5c2c57c"
server_key_version: "encryptionkey@0"
I20260928 17:55:52.744794 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:52.756558 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:52.758487 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:52.758662 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:52.762840 21586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40463
I20260928 17:55:52.762876 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40463 every 8 connection(s)
I20260928 17:55:52.763257 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:52.764189 21654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:52.766407 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b: Bootstrap starting.
I20260928 17:55:52.766923 21654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:52.767146 21654 log.cc:824] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:52.767756 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b: No bootstrap required, opened a new log
I20260928 17:55:52.769500 21654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 17:55:52.769636 21654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:52.769658 21654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c5813af2d44f539eed8931c6867c4b, State: Initialized, Role: FOLLOWER
I20260928 17:55:52.769757 21654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 17:55:52.769834 21654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:52.769869 21654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:52.769958 21654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:52.770066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21586
I20260928 17:55:52.770205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:52.770412 392 external_mini_cluster.cc:1428] Setting key 5d9a602eb7c3f6efe23c46dead41cd8a
I20260928 17:55:52.770689 21654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 17:55:52.770808 21654 leader_election.cc:302] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c5813af2d44f539eed8931c6867c4b; no voters:
I20260928 17:55:52.770968 21654 leader_election.cc:288] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:52.771003 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:52.771165 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [term 1 LEADER]: Becoming Leader. State: Replica: 82c5813af2d44f539eed8931c6867c4b, State: Running, Role: LEADER
I20260928 17:55:52.771219 21654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:52.771246 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } }
I20260928 17:55:52.771597 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c5813af2d44f539eed8931c6867c4b. Latest consensus state: current_term: 1 leader_uuid: "82c5813af2d44f539eed8931c6867c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 17:55:52.771610 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c5813af2d44f539eed8931c6867c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c5813af2d44f539eed8931c6867c4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40463 } } }
I20260928 17:55:52.771746 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:52.771826 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:52.772277 21664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:52.772801 21664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:52.774379 21664 catalog_manager.cc:1347] Generated new cluster ID: b59bdd4db79343a1976cf79aaff1ca46
I20260928 17:55:52.774437 21664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 17:55:52.779865 21664 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:52.780603 21664 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:52.783972 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:52.770906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:36831 (local address 127.0.0.1:40463)
0928 17:55:52.771120 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:52.771129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:52.772942 (+ 1813us) server_negotiation.cc:407] Connection header received
0928 17:55:52.773164 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:52.773171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:52.773231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:52.773338 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:52.774232 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:52.775669 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:52.776767 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:52.777364 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:52.779522 (+ 2158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:52.780251 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:52.780255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:52.780255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:52.780266 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:52.782144 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:52.782542 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:52.782545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:52.782545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:52.782637 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:52.782834 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:52.782834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:52.782834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:52.783107 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:52.783578 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:52.783654 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 17:55:52.783744 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260928 17:55:52.784801 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:52.789754 21664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82c5813af2d44f539eed8931c6867c4b: Generated new TSK 0
I20260928 17:55:52.789916 21664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:52.796380 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:52.789482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:50853 (local address 127.0.0.1:40463)
0928 17:55:52.789532 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:52.789533 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:52.789673 (+ 140us) server_negotiation.cc:407] Connection header received
0928 17:55:52.789725 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:52.789726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:52.789735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:52.789762 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:52.790429 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:52.791193 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:52.792252 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:52.792470 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:52.792527 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:52.793849 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:52.793850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:52.793850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:52.793856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:52.795236 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:52.795643 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:52.795644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:52.795644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:52.795693 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:52.795941 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:52.795942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:52.795942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:52.796032 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:52.796162 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:52.796278 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 17:55:52.796325 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 17:55:52.798280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21586
W20260928 17:55:52.802600 21571 connection.cc:535] client connection to 127.0.0.1:40463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40463): Connection reset by peer (error 104)
2026-09-28T17:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21685](info): set up 2 sockets
Sep 28 17:55:52 dist-test-slave-zqlt krb5kdc[21685](info): commencing operation
krb5kdc: starting...
W20260928 17:55:53.828562 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:53 dist-test-slave-zqlt krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:53.947191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:54.027621 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:54.027865 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:54.027913 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:54.029119 21698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:54.029191 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:54.029217 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:54.029239 21698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:54.029268 21698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:54.030718 21698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:54.030998 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:54.031225 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:54.036015 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:54.036070 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:54.036121 21698 server_base.cc:1034] running on GCE node
W20260928 17:55:54.036070 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:54.036607 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:54.036928 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:54.038089 21698 hybrid_clock.cc:648] HybridClock initialized: now 1790618154038053 us; error 50 us; skew 500 ppm
Sep 28 17:55:54 dist-test-slave-zqlt krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:54.040792 21698 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:54.041283 21698 webserver.cc:460] Webserver started at http://127.0.0.1:35837/ using document root <none> and password file <none>
I20260928 17:55:54.041530 21698 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:54.041577 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:54.041735 21698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:54.042805 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "a7ff66c2d4dd4524b4b54072c70e15c5"
format_stamp: "Formatted at 2026-09-28 17:55:54 on dist-test-slave-zqlt"
server_key: "7bfcf2f3ff3a4bbdc881c23fa0753b28"
server_key_iv: "d5ef8715c018d0b5c29e1dc4edadd4d7"
server_key_version: "encryptionkey@0"
I20260928 17:55:54.043213 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7ff66c2d4dd4524b4b54072c70e15c5"
format_stamp: "Formatted at 2026-09-28 17:55:54 on dist-test-slave-zqlt"
server_key: "7bfcf2f3ff3a4bbdc881c23fa0753b28"
server_key_iv: "d5ef8715c018d0b5c29e1dc4edadd4d7"
server_key_version: "encryptionkey@0"
I20260928 17:55:54.044659 21698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:54.045501 21713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:54.045745 21698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:54.045843 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "a7ff66c2d4dd4524b4b54072c70e15c5"
format_stamp: "Formatted at 2026-09-28 17:55:54 on dist-test-slave-zqlt"
server_key: "7bfcf2f3ff3a4bbdc881c23fa0753b28"
server_key_iv: "d5ef8715c018d0b5c29e1dc4edadd4d7"
server_key_version: "encryptionkey@0"
I20260928 17:55:54.045938 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:54.070691 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:54.072914 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:54.073071 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:54.077509 21698 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43187
I20260928 17:55:54.077584 21765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43187 every 8 connection(s)
I20260928 17:55:54.077960 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:54.079080 21766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:54.081686 21766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: Bootstrap starting.
I20260928 17:55:54.082233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21698
I20260928 17:55:54.082335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:54.082475 21766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:54.082590 392 external_mini_cluster.cc:1428] Setting key 51d6d8d9d5106197e2abe8158a5f1102
I20260928 17:55:54.082845 21766 log.cc:824] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:54.083699 21766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: No bootstrap required, opened a new log
I20260928 17:55:54.086248 21766 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260928 17:55:54.086458 21766 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:54.086493 21766 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ff66c2d4dd4524b4b54072c70e15c5, State: Initialized, Role: FOLLOWER
I20260928 17:55:54.086647 21766 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260928 17:55:54.086755 21766 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:54.086808 21766 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:54.086864 21766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:54.087567 21766 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260928 17:55:54.087697 21766 leader_election.cc:302] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7ff66c2d4dd4524b4b54072c70e15c5; no voters:
I20260928 17:55:54.087898 21766 leader_election.cc:288] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:54.088124 21772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:54.088233 21766 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:54.088341 21772 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [term 1 LEADER]: Becoming Leader. State: Replica: a7ff66c2d4dd4524b4b54072c70e15c5, State: Running, Role: LEADER
I20260928 17:55:54.088497 21772 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260928 17:55:54.088955 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ff66c2d4dd4524b4b54072c70e15c5. Latest consensus state: current_term: 1 leader_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20260928 17:55:54.088985 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ff66c2d4dd4524b4b54072c70e15c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20260928 17:55:54.089054 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:54.083193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:45583 (local address 127.0.0.1:43187)
0928 17:55:54.083382 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:54.083392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:54.083429 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:55:54.083964 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:54.083972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:54.084029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:54.084227 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:54.084490 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:54.085418 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:54.086766 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:54.087315 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:54.087436 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:54.088317 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:54.088322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:54.088322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:54.088561 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:54.088600 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:54.088616 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:54.088698 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:54.088789 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 17:55:54.088820 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260928 17:55:54.089210 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:54.089121 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:54.090178 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 17:55:54.090396 21786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:55:54.090467 21786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:55:54.090555 21780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:54.090742 21780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:54.092339 21780 catalog_manager.cc:1347] Generated new cluster ID: 640e34e524b14b2399fd2f587e1a7c24
I20260928 17:55:54.092401 21780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:54.099320 21780 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:54.099803 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:54.092498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:51147 (local address 127.0.0.1:43187)
0928 17:55:54.092540 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:54.092542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:54.096392 (+ 3850us) server_negotiation.cc:407] Connection header received
0928 17:55:54.096471 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:54.096472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:54.096483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:54.096546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:54.096732 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:54.097575 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:54.098378 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:54.098632 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:54.098716 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:54.099545 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:54.099546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:54.099546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:54.099551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:54.099565 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:54.099569 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:54.099632 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:54.099712 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 17:55:54.099721 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:55:54.100024 21780 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:54.110659 21780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7ff66c2d4dd4524b4b54072c70e15c5: Generated new TSK 0
I20260928 17:55:54.110836 21780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:54.118126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21698
W20260928 17:55:54.122332 21683 connection.cc:535] client connection to 127.0.0.1:43187 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43187): Connection reset by peer (error 104)
2026-09-28T17:55:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:54 dist-test-slave-zqlt krb5kdc[21797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:54 dist-test-slave-zqlt krb5kdc[21797](info): set up 2 sockets
Sep 28 17:55:54 dist-test-slave-zqlt krb5kdc[21797](info): commencing operation
krb5kdc: starting...
W20260928 17:55:55.160319 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:55 dist-test-slave-zqlt krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 17:55:55.276338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:55.357017 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:55.357334 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:55.357375 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:55.358599 21810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:55.358682 21810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:55.358704 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:55.358726 21810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:55.358757 21810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:55.360378 21810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:55.360684 21810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:55.360930 21810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:55.366055 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:55.366065 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:55.366084 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:55.366338 21810 server_base.cc:1034] running on GCE node
I20260928 17:55:55.366561 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:55.366855 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:55.368043 21810 hybrid_clock.cc:648] HybridClock initialized: now 1790618155368033 us; error 32 us; skew 500 ppm
Sep 28 17:55:55 dist-test-slave-zqlt krb5kdc[21797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:55.374315 21810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 17:55:55.374815 21810 webserver.cc:460] Webserver started at http://127.0.0.1:46295/ using document root <none> and password file <none>
I20260928 17:55:55.375025 21810 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:55.375067 21810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:55.375160 21810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:55.376286 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "cf6b19c6eb87472cbcb58bbfd555b450"
format_stamp: "Formatted at 2026-09-28 17:55:55 on dist-test-slave-zqlt"
server_key: "3e08cc0957f3501e535c580db3262dc0"
server_key_iv: "01f03be2c80a79936141745ce9dbbf62"
server_key_version: "encryptionkey@0"
I20260928 17:55:55.376612 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf6b19c6eb87472cbcb58bbfd555b450"
format_stamp: "Formatted at 2026-09-28 17:55:55 on dist-test-slave-zqlt"
server_key: "3e08cc0957f3501e535c580db3262dc0"
server_key_iv: "01f03be2c80a79936141745ce9dbbf62"
server_key_version: "encryptionkey@0"
I20260928 17:55:55.377925 21810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:55.378618 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:55.378809 21810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:55:55.378899 21810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "cf6b19c6eb87472cbcb58bbfd555b450"
format_stamp: "Formatted at 2026-09-28 17:55:55 on dist-test-slave-zqlt"
server_key: "3e08cc0957f3501e535c580db3262dc0"
server_key_iv: "01f03be2c80a79936141745ce9dbbf62"
server_key_version: "encryptionkey@0"
I20260928 17:55:55.378966 21810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:55.382160 21810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:55.383929 21810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:55.384030 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:55.387804 21810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34975
I20260928 17:55:55.387851 21877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34975 every 8 connection(s)
I20260928 17:55:55.388243 21810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:55.389230 21878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:55.390668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21810
I20260928 17:55:55.390797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:55.390998 392 external_mini_cluster.cc:1428] Setting key 1422e6237dd97a3479767227990c07ea
I20260928 17:55:55.391845 21878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450: Bootstrap starting.
I20260928 17:55:55.392542 21878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:55.392874 21878 log.cc:824] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:55.393651 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:55.391478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:46945 (local address 127.0.0.1:34975)
0928 17:55:55.391662 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:55.391672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:55.391705 (+ 33us) server_negotiation.cc:407] Connection header received
0928 17:55:55.392295 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:55.392304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:55.392336 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:55.392499 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:55.393098 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:55.393102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:55.393102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:55.393106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:55.393145 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:55.393165 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:55.393234 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:55.393312 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 17:55:55.393478 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 17:55:55.393664 21878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450: No bootstrap required, opened a new log
I20260928 17:55:55.394680 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:55.395741 21878 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260928 17:55:55.395905 21878 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:55.396052 21878 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6b19c6eb87472cbcb58bbfd555b450, State: Initialized, Role: FOLLOWER
I20260928 17:55:55.396207 21878 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260928 17:55:55.396317 21878 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:55.396351 21878 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:55.396406 21878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:55.396971 21878 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260928 17:55:55.397042 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:55.395974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.169:38239 (local address 127.0.0.1:34975)
0928 17:55:55.396009 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:55.396011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:55.396141 (+ 130us) server_negotiation.cc:407] Connection header received
0928 17:55:55.396210 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:55.396211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:55.396215 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:55.396278 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:55.396843 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:55.396844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:55.396844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:55.396849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:55.396863 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:55.396867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:55.396936 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:55.396972 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 17:55:55.396984 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:55:55.397109 21878 leader_election.cc:302] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6b19c6eb87472cbcb58bbfd555b450; no voters:
I20260928 17:55:55.397274 21878 leader_election.cc:288] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:55.397430 21889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:55.397541 21878 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:55.397589 21889 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [term 1 LEADER]: Becoming Leader. State: Replica: cf6b19c6eb87472cbcb58bbfd555b450, State: Running, Role: LEADER
I20260928 17:55:55.397725 21889 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } }
I20260928 17:55:55.398044 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6b19c6eb87472cbcb58bbfd555b450. Latest consensus state: current_term: 1 leader_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260928 17:55:55.398159 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:55.398063 21890 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6b19c6eb87472cbcb58bbfd555b450" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34975 } } }
I20260928 17:55:55.398284 21890 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:55.398437 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:55.399025 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:55.400470 21897 catalog_manager.cc:1347] Generated new cluster ID: a414a843c86d486c8478356605153b1d
I20260928 17:55:55.400528 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:55.413381 21897 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:55.413618 21897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:55.419872 21897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6b19c6eb87472cbcb58bbfd555b450: Generated new TSK 0
I20260928 17:55:55.420154 21897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:55.426541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21810
2026-09-28T17:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:55 dist-test-slave-zqlt krb5kdc[21909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:55 dist-test-slave-zqlt krb5kdc[21909](info): set up 2 sockets
Sep 28 17:55:55 dist-test-slave-zqlt krb5kdc[21909](info): commencing operation
krb5kdc: starting...
W20260928 17:55:56.464398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[21909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:56.576442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:56.660219 21922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:56.660463 21922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:56.660506 21922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:56.661651 21922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:56.661692 21922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:56.661707 21922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:56.661720 21922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:56.661732 21922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:56.663167 21922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:56.663357 21922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:56.663527 21922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:56.668412 21927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:56.668483 21928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:56.668457 21930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:56.668615 21922 server_base.cc:1034] running on GCE node
I20260928 17:55:56.668887 21922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:56.669194 21922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:56.670394 21922 hybrid_clock.cc:648] HybridClock initialized: now 1790618156670338 us; error 59 us; skew 500 ppm
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[21909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:56.673046 21922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:56.673509 21922 webserver.cc:460] Webserver started at http://127.0.98.62:45937/ using document root <none> and password file <none>
I20260928 17:55:56.673776 21922 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:56.673843 21922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:56.674019 21922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:56.675101 21922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "5f92a75852c34e8a98beb85202d0b14e"
format_stamp: "Formatted at 2026-09-28 17:55:56 on dist-test-slave-zqlt"
server_key: "db56a36dc3c72d32cc383a107a41c744"
server_key_iv: "24dff573e9317e2d5c04b53bf229b2af"
server_key_version: "encryptionkey@0"
I20260928 17:55:56.675475 21922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f92a75852c34e8a98beb85202d0b14e"
format_stamp: "Formatted at 2026-09-28 17:55:56 on dist-test-slave-zqlt"
server_key: "db56a36dc3c72d32cc383a107a41c744"
server_key_iv: "24dff573e9317e2d5c04b53bf229b2af"
server_key_version: "encryptionkey@0"
I20260928 17:55:56.676885 21922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:55:56.677771 21937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:56.678076 21922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:55:56.678184 21922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "5f92a75852c34e8a98beb85202d0b14e"
format_stamp: "Formatted at 2026-09-28 17:55:56 on dist-test-slave-zqlt"
server_key: "db56a36dc3c72d32cc383a107a41c744"
server_key_iv: "24dff573e9317e2d5c04b53bf229b2af"
server_key_version: "encryptionkey@0"
I20260928 17:55:56.678264 21922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:56.688490 21922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:56.690537 21922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:56.690711 21922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:56.694794 21922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46769
I20260928 17:55:56.694832 21989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46769 every 8 connection(s)
I20260928 17:55:56.695220 21922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:56.696157 21990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:56.698355 21990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e: Bootstrap starting.
I20260928 17:55:56.698861 21990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:56.699087 21990 log.cc:824] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:56.699707 21990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e: No bootstrap required, opened a new log
I20260928 17:55:56.701193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 21922
I20260928 17:55:56.701318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:56.701512 392 external_mini_cluster.cc:1428] Setting key f17c8947e9ed0718e612103a506bed6e
I20260928 17:55:56.701658 21990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260928 17:55:56.701825 21990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:56.701860 21990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f92a75852c34e8a98beb85202d0b14e, State: Initialized, Role: FOLLOWER
I20260928 17:55:56.702039 21990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260928 17:55:56.702160 21990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:56.702210 21990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:56.702267 21990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:56.703117 21990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260928 17:55:56.703238 21990 leader_election.cc:302] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f92a75852c34e8a98beb85202d0b14e; no voters:
I20260928 17:55:56.703426 21990 leader_election.cc:288] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:56.703452 21995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:56.703568 21995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [term 1 LEADER]: Becoming Leader. State: Replica: 5f92a75852c34e8a98beb85202d0b14e, State: Running, Role: LEADER
I20260928 17:55:56.703693 21990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:56.703699 21995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260928 17:55:56.704123 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f92a75852c34e8a98beb85202d0b14e. Latest consensus state: current_term: 1 leader_uuid: "5f92a75852c34e8a98beb85202d0b14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } }
I20260928 17:55:56.704217 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f92a75852c34e8a98beb85202d0b14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f92a75852c34e8a98beb85202d0b14e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } }
I20260928 17:55:56.704253 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:56.704312 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:56.704591 22003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:56.705356 22003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:56.707331 22003 catalog_manager.cc:1347] Generated new cluster ID: d099489f009f435ab6409d8744eeefb8
I20260928 17:55:56.707391 22003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[21909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:55:56.712989 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:56.701977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48719 (local address 127.0.98.62:46769)
0928 17:55:56.702150 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:56.702160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:56.702206 (+ 46us) server_negotiation.cc:407] Connection header received
0928 17:55:56.702345 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:56.702352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:56.702408 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:56.702537 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:56.703189 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:56.704515 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:56.705649 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:56.706198 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:56.706340 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:56.709257 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:56.709261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:56.709261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:56.709274 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:56.711231 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:56.711751 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:56.711754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:56.711754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:56.711867 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:56.712119 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:56.712119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:56.712119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:56.712404 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:56.712575 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:56.712699 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:55:56.712795 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 17:55:56.713989 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:56.722993 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:56.715461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37843 (local address 127.0.98.62:46769)
0928 17:55:56.715501 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:56.715503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:56.715688 (+ 185us) server_negotiation.cc:407] Connection header received
0928 17:55:56.715745 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:56.715746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:56.715756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:56.715808 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:56.716462 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:56.717179 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:56.718206 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:56.718571 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:56.718627 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:56.720057 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:56.720058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:56.720058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:55:56.720066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:56.721710 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:56.722149 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:56.722150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:56.722150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:56.722205 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:55:56.722436 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:55:56.722436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:55:56.722436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:55:56.722582 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:56.722677 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:56.722841 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 17:55:56.722901 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:55:56.726718 22003 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:56.727479 22003 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:56.735334 22003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f92a75852c34e8a98beb85202d0b14e: Generated new TSK 0
I20260928 17:55:56.735563 22003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:56.738349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 21922
W20260928 17:55:56.743673 21907 connection.cc:535] client connection to 127.0.98.62:46769 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46769): Connection reset by peer (error 104)
2026-09-28T17:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[22021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[22021](info): set up 2 sockets
Sep 28 17:55:56 dist-test-slave-zqlt krb5kdc[22021](info): commencing operation
krb5kdc: starting...
W20260928 17:55:57.848516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:57 dist-test-slave-zqlt krb5kdc[22021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:57.966530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:58.049283 22034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:58.049535 22034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:58.049578 22034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:58.050753 22034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:58.050827 22034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:58.050855 22034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:58.050879 22034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:58.050912 22034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:58.052472 22034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:58.052752 22034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:58.053009 22034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:58.057922 22042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:58.057968 22040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:58.057919 22039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:58.058259 22034 server_base.cc:1034] running on GCE node
I20260928 17:55:58.058477 22034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:58.058714 22034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:58.059898 22034 hybrid_clock.cc:648] HybridClock initialized: now 1790618158059892 us; error 38 us; skew 500 ppm
Sep 28 17:55:58 dist-test-slave-zqlt krb5kdc[22021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:58.062703 22034 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:58.063175 22034 webserver.cc:460] Webserver started at http://127.0.98.62:45653/ using document root <none> and password file <none>
I20260928 17:55:58.063429 22034 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:58.063498 22034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:58.063647 22034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:58.064774 22034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "772cee35d72c4686bb31bc1751c5aee8"
format_stamp: "Formatted at 2026-09-28 17:55:58 on dist-test-slave-zqlt"
server_key: "c8f3e0f1c2c948ee1fb4cf82ca140fdb"
server_key_iv: "560a907cf02600bcb6e25de591466871"
server_key_version: "encryptionkey@0"
I20260928 17:55:58.065152 22034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "772cee35d72c4686bb31bc1751c5aee8"
format_stamp: "Formatted at 2026-09-28 17:55:58 on dist-test-slave-zqlt"
server_key: "c8f3e0f1c2c948ee1fb4cf82ca140fdb"
server_key_iv: "560a907cf02600bcb6e25de591466871"
server_key_version: "encryptionkey@0"
I20260928 17:55:58.066504 22034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:55:58.067278 22049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:58.067476 22034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:58.067582 22034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "772cee35d72c4686bb31bc1751c5aee8"
format_stamp: "Formatted at 2026-09-28 17:55:58 on dist-test-slave-zqlt"
server_key: "c8f3e0f1c2c948ee1fb4cf82ca140fdb"
server_key_iv: "560a907cf02600bcb6e25de591466871"
server_key_version: "encryptionkey@0"
I20260928 17:55:58.067656 22034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:58.092559 22034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:58.094633 22034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:58.094808 22034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:58.098645 22034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37407
I20260928 17:55:58.098676 22101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37407 every 8 connection(s)
I20260928 17:55:58.099056 22034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:58.099933 22102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:58.101712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22034
I20260928 17:55:58.101810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:58.102015 392 external_mini_cluster.cc:1428] Setting key e2d9cadbe8e362c4359ee5a8e03e25f1
I20260928 17:55:58.102521 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: Bootstrap starting.
I20260928 17:55:58.103215 22102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:58.103531 22102 log.cc:824] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:58.104360 22102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: No bootstrap required, opened a new log
I20260928 17:55:58.106576 22102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } }
I20260928 17:55:58.106740 22102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:58.106776 22102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772cee35d72c4686bb31bc1751c5aee8, State: Initialized, Role: FOLLOWER
I20260928 17:55:58.106900 22102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } }
I20260928 17:55:58.106993 22102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:58.107028 22102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:58.107077 22102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:58.107816 22102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } }
I20260928 17:55:58.107947 22102 leader_election.cc:302] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 772cee35d72c4686bb31bc1751c5aee8; no voters:
I20260928 17:55:58.108135 22102 leader_election.cc:288] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:58.108178 22107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:58.108460 22107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [term 1 LEADER]: Becoming Leader. State: Replica: 772cee35d72c4686bb31bc1751c5aee8, State: Running, Role: LEADER
I20260928 17:55:58.108495 22102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:58.108522 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:58.102560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49907 (local address 127.0.98.62:37407)
0928 17:55:58.102724 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:58.102733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:58.102764 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:55:58.103395 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:58.103405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:58.103480 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:58.103635 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:58.103888 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:58.105337 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:58.106163 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:58.106799 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:58.106905 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:58.107681 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:58.107686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:58.107686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:58.107897 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:58.107936 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:58.107950 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:58.108045 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:58.108207 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 17:55:58.108236 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260928 17:55:58.109196 22107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } }
I20260928 17:55:58.109500 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:58.109867 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "772cee35d72c4686bb31bc1751c5aee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } } }
I20260928 17:55:58.110019 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 772cee35d72c4686bb31bc1751c5aee8. Latest consensus state: current_term: 1 leader_uuid: "772cee35d72c4686bb31bc1751c5aee8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772cee35d72c4686bb31bc1751c5aee8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37407 } } }
I20260928 17:55:58.110955 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:58.110965 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8 [sys.catalog]: This master's current role is: LEADER
W20260928 17:55:58.110275 22121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:55:58.111146 22121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:55:58.111356 22129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:58.111553 22129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:58.113029 22129 catalog_manager.cc:1347] Generated new cluster ID: 15b7abc3e6ee4ee6abf4a7e7c0ce1e6f
I20260928 17:55:58.113075 22129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:58.114231 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:58.111241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47977 (local address 127.0.98.62:37407)
0928 17:55:58.111275 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:58.111277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:58.111285 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:55:58.111303 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:58.111304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:58.111312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:58.111358 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:58.111484 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:58.112025 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:58.112932 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:55:58.113211 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:55:58.113256 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:55:58.113984 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:58.113985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:58.113985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:58.113990 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:58.114004 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:58.114009 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:58.114081 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:58.114153 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 17:55:58.114163 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:55:58.130592 22129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:58.131211 22129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:58.134615 22129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 772cee35d72c4686bb31bc1751c5aee8: Generated new TSK 0
I20260928 17:55:58.134835 22129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:58.137836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22034
W20260928 17:55:58.142340 22019 connection.cc:535] client connection to 127.0.98.62:37407 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37407): Connection reset by peer (error 104)
2026-09-28T17:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:58 dist-test-slave-zqlt krb5kdc[22133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:58 dist-test-slave-zqlt krb5kdc[22133](info): set up 2 sockets
Sep 28 17:55:58 dist-test-slave-zqlt krb5kdc[22133](info): commencing operation
krb5kdc: starting...
W20260928 17:55:59.192348 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:55:59 dist-test-slave-zqlt krb5kdc[22133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:55:59.310057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:55:59.391203 22146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:55:59.391404 22146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:55:59.391426 22146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:55:59.392561 22146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:55:59.392601 22146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:55:59.392616 22146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:55:59.392627 22146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:55:59.392639 22146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:55:59.394006 22146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:55:59.394184 22146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:55:59.394380 22146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:55:59.398980 22151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:59.399017 22152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:55:59.399044 22154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:55:59.399281 22146 server_base.cc:1034] running on GCE node
I20260928 17:55:59.399513 22146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:55:59.399787 22146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:55:59.400949 22146 hybrid_clock.cc:648] HybridClock initialized: now 1790618159400941 us; error 33 us; skew 500 ppm
Sep 28 17:55:59 dist-test-slave-zqlt krb5kdc[22133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:55:59.403489 22146 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:55:59.404001 22146 webserver.cc:460] Webserver started at http://127.0.98.62:32921/ using document root <none> and password file <none>
I20260928 17:55:59.404256 22146 fs_manager.cc:359] Metadata directory not provided
I20260928 17:55:59.404327 22146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:55:59.404495 22146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:55:59.405563 22146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "c337a0d4d39d4845985ef69d1ca77af6"
format_stamp: "Formatted at 2026-09-28 17:55:59 on dist-test-slave-zqlt"
server_key: "d0593b0804bc80b92d03ce8a25153a42"
server_key_iv: "a09a978409819c614d7129e399313290"
server_key_version: "encryptionkey@0"
I20260928 17:55:59.405954 22146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "c337a0d4d39d4845985ef69d1ca77af6"
format_stamp: "Formatted at 2026-09-28 17:55:59 on dist-test-slave-zqlt"
server_key: "d0593b0804bc80b92d03ce8a25153a42"
server_key_iv: "a09a978409819c614d7129e399313290"
server_key_version: "encryptionkey@0"
I20260928 17:55:59.407317 22146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:55:59.408166 22161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:55:59.408351 22146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:55:59.408468 22146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "c337a0d4d39d4845985ef69d1ca77af6"
format_stamp: "Formatted at 2026-09-28 17:55:59 on dist-test-slave-zqlt"
server_key: "d0593b0804bc80b92d03ce8a25153a42"
server_key_iv: "a09a978409819c614d7129e399313290"
server_key_version: "encryptionkey@0"
I20260928 17:55:59.408566 22146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:55:59.412001 22146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:55:59.414518 22146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:55:59.414692 22146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:55:59.418691 22146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45821
I20260928 17:55:59.418725 22213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45821 every 8 connection(s)
I20260928 17:55:59.419101 22146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:55:59.420707 22214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:55:59.422873 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6: Bootstrap starting.
I20260928 17:55:59.423382 22214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6: Neither blocks nor log segments found. Creating new log.
I20260928 17:55:59.423619 22214 log.cc:824] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6: Log is configured to *not* fsync() on all Append() calls
I20260928 17:55:59.424330 22214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6: No bootstrap required, opened a new log
I20260928 17:55:59.424947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22146
I20260928 17:55:59.425058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:55:59.425242 392 external_mini_cluster.cc:1428] Setting key fa7311222e96aa930729e4a00f3f1068
I20260928 17:55:59.426779 22214 raft_consensus.cc:348] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } }
I20260928 17:55:59.426934 22214 raft_consensus.cc:374] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:55:59.426996 22214 raft_consensus.cc:729] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c337a0d4d39d4845985ef69d1ca77af6, State: Initialized, Role: FOLLOWER
I20260928 17:55:59.427129 22214 consensus_queue.cc:260] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } }
I20260928 17:55:59.427225 22214 raft_consensus.cc:388] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:55:59.427280 22214 raft_consensus.cc:482] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:55:59.427331 22214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:55:59.428259 22214 raft_consensus.cc:504] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } }
I20260928 17:55:59.428375 22214 leader_election.cc:302] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c337a0d4d39d4845985ef69d1ca77af6; no voters:
I20260928 17:55:59.428412 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:59.425742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58393 (local address 127.0.98.62:45821)
0928 17:55:59.425924 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:59.425943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:59.425978 (+ 35us) server_negotiation.cc:407] Connection header received
0928 17:55:59.426803 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:59.426811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:59.426843 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:59.427035 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:59.427856 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:59.427860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:59.427860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:59.427865 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:59.427927 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:59.427943 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:59.428032 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:59.428169 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 17:55:59.428214 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 17:55:59.428586 22214 leader_election.cc:288] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:55:59.428612 22219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:55:59.428721 22219 raft_consensus.cc:686] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [term 1 LEADER]: Becoming Leader. State: Replica: c337a0d4d39d4845985ef69d1ca77af6, State: Running, Role: LEADER
I20260928 17:55:59.428834 22219 consensus_queue.cc:237] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } }
I20260928 17:55:59.428843 22214 sys_catalog.cc:564] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:55:59.429214 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c337a0d4d39d4845985ef69d1ca77af6. Latest consensus state: current_term: 1 leader_uuid: "c337a0d4d39d4845985ef69d1ca77af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } } }
I20260928 17:55:59.429327 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:59.429486 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:55:59.429567 22223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:55:59.430104 22223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:55:59.430727 22221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c337a0d4d39d4845985ef69d1ca77af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c337a0d4d39d4845985ef69d1ca77af6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45821 } } }
I20260928 17:55:59.430881 22221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6 [sys.catalog]: This master's current role is: LEADER
I20260928 17:55:59.432468 22223 catalog_manager.cc:1347] Generated new cluster ID: 5af3e3d0851343b8a0a34429cdeccd9c
I20260928 17:55:59.432536 22223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:55:59.432615 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:55:59.431209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58129 (local address 127.0.98.62:45821)
0928 17:55:59.431254 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:55:59.431256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:55:59.431394 (+ 138us) server_negotiation.cc:407] Connection header received
0928 17:55:59.431501 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:55:59.431502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:55:59.431507 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:55:59.431540 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:55:59.432302 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:55:59.432304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:55:59.432304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:55:59.432309 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:55:59.432325 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:55:59.432330 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:55:59.432393 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:55:59.432539 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 17:55:59.432554 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:55:59.445492 22223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:55:59.445708 22223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:55:59.453012 22223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c337a0d4d39d4845985ef69d1ca77af6: Generated new TSK 0
I20260928 17:55:59.453265 22223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:55:59.456869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22146
2026-09-28T17:55:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:55:59 dist-test-slave-zqlt krb5kdc[22245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:55:59 dist-test-slave-zqlt krb5kdc[22245](info): set up 2 sockets
Sep 28 17:55:59 dist-test-slave-zqlt krb5kdc[22245](info): commencing operation
krb5kdc: starting...
W20260928 17:56:00.488441 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:56:00.604588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:56:00.684310 22258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:00.684514 22258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:00.684537 22258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:00.685742 22258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:00.685787 22258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:00.685801 22258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:00.685844 22258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:00.685875 22258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:00.687371 22258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:00.687628 22258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:00.687855 22258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:00.692728 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:00.692724 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:00.692798 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:00.693060 22258 server_base.cc:1034] running on GCE node
I20260928 17:56:00.693300 22258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:00.693559 22258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:00.694741 22258 hybrid_clock.cc:648] HybridClock initialized: now 1790618160694726 us; error 30 us; skew 500 ppm
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:00.701390 22258 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:56:00.701843 22258 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260928 17:56:00.702042 22258 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:00.702081 22258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:00.702181 22258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:00.703159 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "dfbbae04c475437587b5f876fd978983"
format_stamp: "Formatted at 2026-09-28 17:56:00 on dist-test-slave-zqlt"
server_key: "2b0ffc36e160ea2480a2479ea8fc5018"
server_key_iv: "345c6064cdcb2520bce7d903ee23c3dd"
server_key_version: "encryptionkey@0"
I20260928 17:56:00.703457 22258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfbbae04c475437587b5f876fd978983"
format_stamp: "Formatted at 2026-09-28 17:56:00 on dist-test-slave-zqlt"
server_key: "2b0ffc36e160ea2480a2479ea8fc5018"
server_key_iv: "345c6064cdcb2520bce7d903ee23c3dd"
server_key_version: "encryptionkey@0"
I20260928 17:56:00.704757 22258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:56:00.705502 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:00.705736 22258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:56:00.705843 22258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "dfbbae04c475437587b5f876fd978983"
format_stamp: "Formatted at 2026-09-28 17:56:00 on dist-test-slave-zqlt"
server_key: "2b0ffc36e160ea2480a2479ea8fc5018"
server_key_iv: "345c6064cdcb2520bce7d903ee23c3dd"
server_key_version: "encryptionkey@0"
I20260928 17:56:00.705902 22258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:00.744854 22258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:00.746910 22258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:00.747035 22258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:00.751127 22258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45503
I20260928 17:56:00.751199 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45503 every 8 connection(s)
I20260928 17:56:00.751579 22258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:56:00.752538 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:00.754820 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983: Bootstrap starting.
I20260928 17:56:00.755398 22326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:00.755682 22326 log.cc:824] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:00.756515 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983: No bootstrap required, opened a new log
I20260928 17:56:00.758054 22326 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260928 17:56:00.758234 22326 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:56:00.758281 22326 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbbae04c475437587b5f876fd978983, State: Initialized, Role: FOLLOWER
I20260928 17:56:00.758450 22326 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260928 17:56:00.758544 22326 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:56:00.758589 22326 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:56:00.758641 22326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:00.759164 22326 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260928 17:56:00.759285 22326 leader_election.cc:302] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbbae04c475437587b5f876fd978983; no voters:
I20260928 17:56:00.759457 22326 leader_election.cc:288] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:56:00.759495 22329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:56:00.759521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22258
I20260928 17:56:00.759634 22329 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [term 1 LEADER]: Becoming Leader. State: Replica: dfbbae04c475437587b5f876fd978983, State: Running, Role: LEADER
I20260928 17:56:00.759680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:56:00.759725 22326 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:00.759707 22329 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260928 17:56:00.759871 392 external_mini_cluster.cc:1428] Setting key 0125d61ccb4ac00eaa886db482d67a32
I20260928 17:56:00.760022 22329 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbbae04c475437587b5f876fd978983. Latest consensus state: current_term: 1 leader_uuid: "dfbbae04c475437587b5f876fd978983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260928 17:56:00.760080 22329 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:00.760315 22333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:56:00.760311 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbbae04c475437587b5f876fd978983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbae04c475437587b5f876fd978983" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260928 17:56:00.760450 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:00.761544 22333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:56:00.763198 22333 catalog_manager.cc:1347] Generated new cluster ID: 076a9260a9af4f85b6c4d16a49ead021
I20260928 17:56:00.763262 22333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:00.770802 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:00.760804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41167 (local address 127.0.98.62:45503)
0928 17:56:00.761002 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:00.761013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:00.761049 (+ 36us) server_negotiation.cc:407] Connection header received
0928 17:56:00.761203 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:00.761212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:00.761293 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:00.761451 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:00.762176 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:00.763535 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:00.764415 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:00.764855 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:00.764976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:00.767367 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:00.767372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:00.767372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:00.767393 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:00.769171 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:00.769561 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:00.769564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:00.769564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:00.769680 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:00.769903 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:00.769904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:00.769904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:00.770163 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:00.770341 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:00.770441 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 17:56:00.770591 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260928 17:56:00.771945 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:56:00.772207 22333 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:56:00.772890 22333 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:56:00.778527 22333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbbae04c475437587b5f876fd978983: Generated new TSK 0
I20260928 17:56:00.778690 22333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:56:00.785054 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:00.773397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60769 (local address 127.0.98.62:45503)
0928 17:56:00.773436 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:00.773437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:00.778290 (+ 4853us) server_negotiation.cc:407] Connection header received
0928 17:56:00.778390 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:00.778391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:00.778404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:00.778461 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:00.779416 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:00.779996 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:00.781080 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:00.781289 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:00.781338 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:00.782688 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:00.782690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:00.782690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:00.782696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:00.784005 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:00.784409 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:00.784410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:00.784410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:00.784461 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:00.784658 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:00.784658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:00.784659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:00.784748 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:00.784817 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:00.784940 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 17:56:00.784999 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 17:56:00.787297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22258
2026-09-28T17:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22357](info): set up 2 sockets
Sep 28 17:56:00 dist-test-slave-zqlt krb5kdc[22357](info): commencing operation
krb5kdc: starting...
W20260928 17:56:01.908253 392 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:56:01 dist-test-slave-zqlt krb5kdc[22357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:56:02.026423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:56:02.108343 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:02.108551 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:02.108573 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:02.109712 22370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:02.109754 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:02.109768 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:02.109781 22370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:02.109836 22370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:02.111363 22370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:02.111558 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:02.111799 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:02.116539 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:02.116544 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:02.116554 22370 server_base.cc:1034] running on GCE node
W20260928 17:56:02.116526 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:02.116932 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:02.117229 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:02.118394 22370 hybrid_clock.cc:648] HybridClock initialized: now 1790618162118383 us; error 43 us; skew 500 ppm
Sep 28 17:56:02 dist-test-slave-zqlt krb5kdc[22357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:02.120924 22370 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:56:02.121351 22370 webserver.cc:460] Webserver started at http://127.0.98.62:36011/ using document root <none> and password file <none>
I20260928 17:56:02.121587 22370 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:02.121657 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:02.121809 22370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:02.122859 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "3cbe0ae817a84743bd680350e7e6ea25"
format_stamp: "Formatted at 2026-09-28 17:56:02 on dist-test-slave-zqlt"
server_key: "c3de597815e001e22bd9d66de046889f"
server_key_iv: "fd123078242e640bb22794d35cc95a10"
server_key_version: "encryptionkey@0"
I20260928 17:56:02.123222 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cbe0ae817a84743bd680350e7e6ea25"
format_stamp: "Formatted at 2026-09-28 17:56:02 on dist-test-slave-zqlt"
server_key: "c3de597815e001e22bd9d66de046889f"
server_key_iv: "fd123078242e640bb22794d35cc95a10"
server_key_version: "encryptionkey@0"
I20260928 17:56:02.124643 22370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 17:56:02.125330 22385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:02.125496 22370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 17:56:02.125597 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "3cbe0ae817a84743bd680350e7e6ea25"
format_stamp: "Formatted at 2026-09-28 17:56:02 on dist-test-slave-zqlt"
server_key: "c3de597815e001e22bd9d66de046889f"
server_key_iv: "fd123078242e640bb22794d35cc95a10"
server_key_version: "encryptionkey@0"
I20260928 17:56:02.125687 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:02.150302 22370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:02.152274 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:02.152383 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:02.156109 22370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44421
I20260928 17:56:02.156188 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44421 every 8 connection(s)
I20260928 17:56:02.156531 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:56:02.157343 22438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:02.159572 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25: Bootstrap starting.
I20260928 17:56:02.160156 22438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:02.160437 22438 log.cc:824] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:02.161104 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25: No bootstrap required, opened a new log
I20260928 17:56:02.161751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22370
I20260928 17:56:02.161870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:56:02.162068 392 external_mini_cluster.cc:1428] Setting key e9f473523fca2bc801f3fc47ca6ca2b5
I20260928 17:56:02.163370 22438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } }
I20260928 17:56:02.163527 22438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:56:02.163549 22438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbe0ae817a84743bd680350e7e6ea25, State: Initialized, Role: FOLLOWER
I20260928 17:56:02.163683 22438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } }
I20260928 17:56:02.163769 22438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:56:02.163801 22438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:56:02.163902 22438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:02.164614 22438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } }
I20260928 17:56:02.164736 22438 leader_election.cc:302] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cbe0ae817a84743bd680350e7e6ea25; no voters:
I20260928 17:56:02.164904 22438 leader_election.cc:288] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:56:02.164963 22443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:56:02.165134 22438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:02.165148 22443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [term 1 LEADER]: Becoming Leader. State: Replica: 3cbe0ae817a84743bd680350e7e6ea25, State: Running, Role: LEADER
I20260928 17:56:02.165338 22443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } }
I20260928 17:56:02.165795 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cbe0ae817a84743bd680350e7e6ea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } } }
I20260928 17:56:02.166092 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:02.166025 22445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cbe0ae817a84743bd680350e7e6ea25. Latest consensus state: current_term: 1 leader_uuid: "3cbe0ae817a84743bd680350e7e6ea25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cbe0ae817a84743bd680350e7e6ea25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44421 } } }
I20260928 17:56:02.166568 22445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:02.166564 22456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:56:02.168017 22456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:56:02.168988 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:02.162559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54029 (local address 127.0.98.62:44421)
0928 17:56:02.162747 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:02.162754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:02.162779 (+ 25us) server_negotiation.cc:407] Connection header received
0928 17:56:02.163513 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:02.163521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:02.163615 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:02.163783 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:02.164053 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:02.165428 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:02.166702 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:02.167113 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:02.167179 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:02.168145 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:02.168149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:02.168150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:56:02.168368 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:02.168406 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:56:02.168420 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:02.168504 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:02.168711 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 17:56:02.168780 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260928 17:56:02.169718 22456 catalog_manager.cc:1347] Generated new cluster ID: 92e92144f6f447b8a3cb23269042751b
I20260928 17:56:02.169773 22456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:56:02.172401 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:56:02.179821 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:02.175585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53543 (local address 127.0.98.62:44421)
0928 17:56:02.176579 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:02.176581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:02.176590 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:56:02.176611 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:02.176612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:02.176622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:02.176674 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:02.176848 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:02.177550 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:02.178509 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:02.178784 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:02.178817 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:02.179557 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:02.179559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:02.179559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:56:02.179563 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:02.179580 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:56:02.179586 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:02.179658 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:02.179714 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 17:56:02.179739 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973}
I20260928 17:56:02.181643 22456 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:56:02.182135 22456 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:56:02.185774 22456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cbe0ae817a84743bd680350e7e6ea25: Generated new TSK 0
I20260928 17:56:02.186012 22456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:56:02.187752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22370
2026-09-28T17:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:56:02 dist-test-slave-zqlt krb5kdc[22469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:56:02 dist-test-slave-zqlt krb5kdc[22469](info): set up 2 sockets
Sep 28 17:56:02 dist-test-slave-zqlt krb5kdc[22469](info): commencing operation
krb5kdc: starting...
W20260928 17:56:03.260489 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:56:03 dist-test-slave-zqlt krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:56:03.382186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:56:03.461951 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:03.462172 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:03.462193 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:03.463285 22482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:03.463321 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:03.463335 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:03.463348 22482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:03.463361 22482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:03.464809 22482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:03.465000 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:03.465173 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:03.470109 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:03.470115 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:03.470119 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:03.470433 22482 server_base.cc:1034] running on GCE node
I20260928 17:56:03.470650 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:03.470950 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:03.472149 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790618163472130 us; error 40 us; skew 500 ppm
Sep 28 17:56:03 dist-test-slave-zqlt krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:03.474743 22482 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:56:03.475251 22482 webserver.cc:460] Webserver started at http://127.0.98.62:41713/ using document root <none> and password file <none>
I20260928 17:56:03.475500 22482 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:03.475574 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:03.475728 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:03.476950 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "506b2e0711244d06b951b7fbd50151e6"
format_stamp: "Formatted at 2026-09-28 17:56:03 on dist-test-slave-zqlt"
server_key: "0bec833c20e65bb3b171ad6dd673c361"
server_key_iv: "1f740becf2f6d35e0cf826327dc7060b"
server_key_version: "encryptionkey@0"
I20260928 17:56:03.477350 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "506b2e0711244d06b951b7fbd50151e6"
format_stamp: "Formatted at 2026-09-28 17:56:03 on dist-test-slave-zqlt"
server_key: "0bec833c20e65bb3b171ad6dd673c361"
server_key_iv: "1f740becf2f6d35e0cf826327dc7060b"
server_key_version: "encryptionkey@0"
I20260928 17:56:03.478739 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:56:03.479507 22497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:03.479703 22482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:56:03.479812 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "506b2e0711244d06b951b7fbd50151e6"
format_stamp: "Formatted at 2026-09-28 17:56:03 on dist-test-slave-zqlt"
server_key: "0bec833c20e65bb3b171ad6dd673c361"
server_key_iv: "1f740becf2f6d35e0cf826327dc7060b"
server_key_version: "encryptionkey@0"
I20260928 17:56:03.479892 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:03.483332 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:03.485867 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:03.486032 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:03.490391 22482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38259
I20260928 17:56:03.490420 22549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38259 every 8 connection(s)
I20260928 17:56:03.490785 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:56:03.491773 22550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:03.494095 22550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6: Bootstrap starting.
I20260928 17:56:03.494633 22550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:03.494861 22550 log.cc:824] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:03.495505 22550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6: No bootstrap required, opened a new log
I20260928 17:56:03.496933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22482
I20260928 17:56:03.497093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:56:03.497285 392 external_mini_cluster.cc:1428] Setting key 21c6a9160acc71999b5b8747fc59e94b
I20260928 17:56:03.497188 22550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260928 17:56:03.497339 22550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 17:56:03.497400 22550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b2e0711244d06b951b7fbd50151e6, State: Initialized, Role: FOLLOWER
I20260928 17:56:03.497551 22550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260928 17:56:03.497665 22550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 17:56:03.497730 22550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 17:56:03.497786 22550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:03.498404 22550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260928 17:56:03.498531 22550 leader_election.cc:302] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506b2e0711244d06b951b7fbd50151e6; no voters:
I20260928 17:56:03.498716 22550 leader_election.cc:288] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 17:56:03.498761 22555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:56:03.498906 22555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [term 1 LEADER]: Becoming Leader. State: Replica: 506b2e0711244d06b951b7fbd50151e6, State: Running, Role: LEADER
I20260928 17:56:03.498975 22550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:03.499115 22555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } }
I20260928 17:56:03.499593 22556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506b2e0711244d06b951b7fbd50151e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } } }
I20260928 17:56:03.499701 22556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:03.499953 22564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:56:03.500290 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:03.497811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48571 (local address 127.0.98.62:38259)
0928 17:56:03.498020 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:03.498031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:03.498068 (+ 37us) server_negotiation.cc:407] Connection header received
0928 17:56:03.498392 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:03.498398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:03.498423 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:03.498591 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:03.499696 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:03.499700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:03.499700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:56:03.499704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:03.499752 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:56:03.499766 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:03.499829 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:03.500022 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 17:56:03.500070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260928 17:56:03.500602 22557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 506b2e0711244d06b951b7fbd50151e6. Latest consensus state: current_term: 1 leader_uuid: "506b2e0711244d06b951b7fbd50151e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506b2e0711244d06b951b7fbd50151e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38259 } } }
I20260928 17:56:03.500684 22557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:03.500744 22564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:56:03.501262 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:56:03.503631 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:03.502566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48357 (local address 127.0.98.62:38259)
0928 17:56:03.502597 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:03.502598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:03.502740 (+ 142us) server_negotiation.cc:407] Connection header received
0928 17:56:03.502797 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:03.502797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:03.502801 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:03.502848 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:03.503427 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:03.503428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:03.503428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 17:56:03.503432 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:03.503448 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 17:56:03.503452 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:03.503502 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:03.503581 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 17:56:03.503591 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 17:56:03.505081 22564 catalog_manager.cc:1347] Generated new cluster ID: cf59dcbb20d641beb1f3975f33cd810b
I20260928 17:56:03.505132 22564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:56:03.514552 22564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:56:03.514750 22564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:56:03.523898 22564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 506b2e0711244d06b951b7fbd50151e6: Generated new TSK 0
I20260928 17:56:03.524194 22564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:56:03.533653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22482
2026-09-28T17:56:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1345 ms)
[----------] 18 tests from AuthTokenIssuingTest (23883 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:56:03 dist-test-slave-zqlt krb5kdc[22581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:56:03 dist-test-slave-zqlt krb5kdc[22581](info): set up 2 sockets
Sep 28 17:56:03 dist-test-slave-zqlt krb5kdc[22581](info): commencing operation
krb5kdc: starting...
W20260928 17:56:04.560375 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:56:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:56:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:56:04.679476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:56:04.762095 22594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:04.762329 22594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:04.762348 22594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:04.762370 22594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:04.763569 22594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:04.763609 22594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:04.763624 22594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:04.763636 22594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:04.763649 22594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:04.765158 22594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:04.765375 22594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:04.765573 22594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:04.770533 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:04.770533 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:04.770622 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:04.770951 22594 server_base.cc:1034] running on GCE node
I20260928 17:56:04.771212 22594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:04.771502 22594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:04.772673 22594 hybrid_clock.cc:648] HybridClock initialized: now 1790618164772665 us; error 45 us; skew 500 ppm
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:04.775215 22594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:56:04.775717 22594 webserver.cc:460] Webserver started at http://127.0.98.62:41379/ using document root <none> and password file <none>
I20260928 17:56:04.775997 22594 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:04.776072 22594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:04.776242 22594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:04.777345 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "4188e7d08d7f4921beac3fbd3a574393"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "36a81611928542968823732ef8b3bc0c"
server_key_iv: "bb65e03904e9202386d9a64daffb89a5"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.777725 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "4188e7d08d7f4921beac3fbd3a574393"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "36a81611928542968823732ef8b3bc0c"
server_key_iv: "bb65e03904e9202386d9a64daffb89a5"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.779130 22594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:56:04.779954 22609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:04.780225 22594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:56:04.780334 22594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "4188e7d08d7f4921beac3fbd3a574393"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "36a81611928542968823732ef8b3bc0c"
server_key_iv: "bb65e03904e9202386d9a64daffb89a5"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.780422 22594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:04.788496 22594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:04.790328 22594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:04.790498 22594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:04.794557 22594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42999
I20260928 17:56:04.794602 22661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42999 every 8 connection(s)
I20260928 17:56:04.794950 22594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:56:04.795876 22662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:04.797377 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:04.804409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22594
I20260928 17:56:04.804520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:56:04.804729 392 external_mini_cluster.cc:1428] Setting key 1c823c3bb8af68bca2095904d2999626
I20260928 17:56:04.807346 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.797955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:40624)
0928 17:56:04.798166 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.798208 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:04.798709 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:04.798996 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:04.798999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:04.799275 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:04.799416 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.799421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.800480 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.800481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.801141 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.801143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.801655 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.801656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.801718 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.802241 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:04.802249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:04.803540 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:04.805784 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.805790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.805791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.805988 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.806285 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.806286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.806286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.806372 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.806719 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:04.806724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:04.806783 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 17:56:04.806890 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 17:56:04.807159 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":143,"thread_start_us":86,"threads_started":1}
I20260928 17:56:04.807348 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.798017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.98.62:42999)
0928 17:56:04.798182 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:04.798192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:04.798312 (+ 120us) server_negotiation.cc:407] Connection header received
0928 17:56:04.798870 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:04.798873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:04.798894 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:04.798988 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:04.799518 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.800372 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.801242 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.801537 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.801666 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.803627 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:04.803633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:04.803634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:04.803640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:04.805696 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.806130 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.806132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.806133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.806211 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.806441 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.806442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.806442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.806644 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:04.806742 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:04.806915 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 17:56:04.807115 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260928 17:56:04.808137 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:04.808502 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.808370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50280)
0928 17:56:04.808421 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.808482 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 17:56:04.808667 22612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
I20260928 17:56:04.809046 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.808981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50288)
0928 17:56:04.809007 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.809030 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 17:56:04.809170 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 17:56:04.832393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36733
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:56:04.858388 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } attempt: 1
I20260928 17:56:04.859035 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.858865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50304)
0928 17:56:04.858952 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.859001 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 17:56:04.859416 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.859328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50318)
0928 17:56:04.859372 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.859399 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:56:04.859527 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
W20260928 17:56:04.914382 22669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:04.914680 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:04.914713 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:04.914747 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:04.915982 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:04.916061 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:04.916085 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:04.916103 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:04.916154 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:04.917816 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:04.918094 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:04.918368 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:04.923305 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:04.923374 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:04.923396 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:04.923731 22669 server_base.cc:1034] running on GCE node
I20260928 17:56:04.923985 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:04.924329 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:04.925479 22669 hybrid_clock.cc:648] HybridClock initialized: now 1790618164925471 us; error 37 us; skew 500 ppm
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:04.927693 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } attempt: 2
I20260928 17:56:04.928175 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.927980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50322)
0928 17:56:04.928054 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.928117 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:56:04.928293 22669 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 17:56:04.928565 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.928471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50336)
0928 17:56:04.928517 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.928539 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 17:56:04.928664 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36733: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36733: connect: Connection refused (error 111)
I20260928 17:56:04.928882 22669 webserver.cc:460] Webserver started at http://127.0.98.61:36853/ using document root <none> and password file <none>
I20260928 17:56:04.929131 22669 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:04.929179 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:04.929330 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:04.930437 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data/instance:
uuid: "25f7ca5236364e37a8fa56ab7a419909"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "5b486bead0ba385b6fb492187aee1b68"
server_key_iv: "7ee6022f1dcdfbda1001df6635cfcb74"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.930817 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal/instance:
uuid: "25f7ca5236364e37a8fa56ab7a419909"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "5b486bead0ba385b6fb492187aee1b68"
server_key_iv: "7ee6022f1dcdfbda1001df6635cfcb74"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.932340 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:56:04.933183 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:04.933396 22669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:56:04.933487 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "25f7ca5236364e37a8fa56ab7a419909"
format_stamp: "Formatted at 2026-09-28 17:56:04 on dist-test-slave-zqlt"
server_key: "5b486bead0ba385b6fb492187aee1b68"
server_key_iv: "7ee6022f1dcdfbda1001df6635cfcb74"
server_key_version: "encryptionkey@0"
I20260928 17:56:04.933562 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:04.961946 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:04.964184 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:04.964332 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:04.968562 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36733
I20260928 17:56:04.968619 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36733 every 8 connection(s)
I20260928 17:56:04.968950 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
I20260928 17:56:04.969901 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:04.971487 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:04.977819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22669
I20260928 17:56:04.977897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-1/wal/instance
I20260928 17:56:04.978080 392 external_mini_cluster.cc:1428] Setting key 716241c0fa901271459eb83250c43142
I20260928 17:56:04.981137 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.972187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.98.62:42999)
0928 17:56:04.972255 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:04.972257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:04.972453 (+ 196us) server_negotiation.cc:407] Connection header received
0928 17:56:04.972968 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:04.972970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:04.972982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:04.973051 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:04.973667 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.974192 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.975206 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.975439 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.975537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.977777 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:04.977779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:04.977779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:04.977787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:04.979512 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.980017 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.980018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.980018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.980134 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.980468 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.980469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.980469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.980592 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:04.980657 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:04.980997 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 17:56:04.981071 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 17:56:04.981434 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.972203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:40630)
0928 17:56:04.972333 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.972369 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:04.972850 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:04.973127 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:04.973131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:04.973378 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:04.973570 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.973574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.974306 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.974308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.975111 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.975113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.975556 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.975557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.975704 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.976256 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:04.976264 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:04.977660 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:04.979600 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.979608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.979610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.979929 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.980216 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.980217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.980218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.980374 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.980668 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:04.980675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:04.980859 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 17:56:04.980975 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 17:56:04.981152 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260928 17:56:04.982185 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 17:56:04 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:04.992419 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.982439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50352)
0928 17:56:04.982489 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.982500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:04.982540 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:04.982826 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:04.982827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:04.982994 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:04.983063 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.983065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.983925 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.983926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.984802 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:04.984804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.985267 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.985268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:04.985352 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.985764 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:04.985770 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:04.989191 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:04.991281 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.991282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.991283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.991447 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.991688 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:04.991689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:04.991690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:04.991744 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:04.992046 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:04.992047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:04.992115 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 17:56:04.992192 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 17:56:04.992299 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 17:56:04.992479 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.982533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.0.98.61:36733)
0928 17:56:04.982693 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:04.982698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:04.982714 (+ 16us) server_negotiation.cc:407] Connection header received
0928 17:56:04.982733 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:04.982735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:04.982753 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:04.982808 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:04.983164 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.983864 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.984882 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:04.985184 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:04.985240 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:04.989293 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:04.989303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:04.989304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:04.989311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:04.991185 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.991539 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.991542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.991543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.991601 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:04.991815 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:04.991817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:04.991817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:04.991967 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:04.992056 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:04.992233 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 17:56:04.992351 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":131,"thread_start_us":113,"threads_started":1}
I20260928 17:56:04.993063 22737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:04.993434 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.993313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35890)
0928 17:56:04.993359 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.993402 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:56:04.993614 22685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
I20260928 17:56:04.994055 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:04.993991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35900)
0928 17:56:04.994017 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:04.994039 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 17:56:04.994172 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 17:56:05.004746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33911
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:56:05.019856 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } attempt: 3
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:05.027428 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.020276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50360)
0928 17:56:05.020348 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.020367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.020443 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.020618 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.020619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.020800 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.020863 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.020866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.021595 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.021597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.022160 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.022163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.022556 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.022557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.022642 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.023177 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.023181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.024485 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.026431 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.026432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.026432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.026614 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.026833 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.026834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.026834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.026883 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.027180 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.027180 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.027224 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.027277 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.027356 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 17:56:05.027505 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.020350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.0.98.61:36733)
0928 17:56:05.020436 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.020438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.020451 (+ 13us) server_negotiation.cc:407] Connection header received
0928 17:56:05.020504 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.020506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.020518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.020613 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.020951 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.021491 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.022256 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.022471 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.022553 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.024579 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.024581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.024582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.024591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.026313 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.026693 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.026694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.026694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.026762 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.026946 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.026947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.026948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.027100 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.027186 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.027320 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.027404 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 17:56:05.027964 22662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:05.028352 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.028245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35914)
0928 17:56:05.028296 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.028325 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:56:05.028748 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.028666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35924)
0928 17:56:05.028710 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.028733 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 17:56:05.028858 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
I20260928 17:56:05.043403 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } attempt: 1
I20260928 17:56:05.043921 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.043779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35938)
0928 17:56:05.043845 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.043887 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:56:05.044355 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.044256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35948)
0928 17:56:05.044308 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.044335 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 17:56:05.044456 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
I20260928 17:56:05.060026 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } attempt: 1
I20260928 17:56:05.060577 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.060399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35964)
0928 17:56:05.060480 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.060536 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 17:56:05.061005 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.060900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35978)
0928 17:56:05.060949 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.060983 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 17:56:05.061113 22662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
W20260928 17:56:05.091487 22744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:05.091706 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:05.091725 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:05.091776 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:05.092943 22744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:05.092988 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:05.093003 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:05.093060 22744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:05.093092 22744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:05.094580 22744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42999,127.0.98.61:36733,127.0.98.60:33911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:05.094844 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:05.095085 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 17:56:05.099987 22744 server_base.cc:1034] running on GCE node
W20260928 17:56:05.099943 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:05.099995 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:05.099956 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:05.100436 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:05.100731 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:05.101898 22744 hybrid_clock.cc:648] HybridClock initialized: now 1790618165101878 us; error 39 us; skew 500 ppm
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:05.104449 22744 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 17:56:05.104893 22744 webserver.cc:460] Webserver started at http://127.0.98.60:38073/ using document root <none> and password file <none>
I20260928 17:56:05.105130 22744 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:05.105204 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:05.105365 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:05.106453 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data/instance:
uuid: "ed0b99f113504394bb0d16ef922bba11"
format_stamp: "Formatted at 2026-09-28 17:56:05 on dist-test-slave-zqlt"
server_key: "6ee86374c4a06a9bcd3f88130321a252"
server_key_iv: "7831d82c37339106050693e6f159edc3"
server_key_version: "encryptionkey@0"
I20260928 17:56:05.106837 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal/instance:
uuid: "ed0b99f113504394bb0d16ef922bba11"
format_stamp: "Formatted at 2026-09-28 17:56:05 on dist-test-slave-zqlt"
server_key: "6ee86374c4a06a9bcd3f88130321a252"
server_key_iv: "7831d82c37339106050693e6f159edc3"
server_key_version: "encryptionkey@0"
I20260928 17:56:05.108280 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:56:05.109016 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:05.109192 22744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:56:05.109298 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "ed0b99f113504394bb0d16ef922bba11"
format_stamp: "Formatted at 2026-09-28 17:56:05 on dist-test-slave-zqlt"
server_key: "6ee86374c4a06a9bcd3f88130321a252"
server_key_iv: "7831d82c37339106050693e6f159edc3"
server_key_version: "encryptionkey@0"
I20260928 17:56:05.109400 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:05.112603 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } attempt: 2
I20260928 17:56:05.113005 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.112876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35980)
0928 17:56:05.112935 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.112971 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:56:05.113348 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.113266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35988)
0928 17:56:05.113307 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.113331 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 17:56:05.113451 22737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33911: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33911: connect: Connection refused (error 111)
I20260928 17:56:05.125434 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:05.127767 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:05.127882 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:05.131963 22744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33911
I20260928 17:56:05.132038 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33911 every 8 connection(s)
I20260928 17:56:05.132411 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
I20260928 17:56:05.133430 22812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:05.135049 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:05.136309 22662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } attempt: 2
I20260928 17:56:05.139896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 22744
I20260928 17:56:05.140002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790617898245922-392-0/minicluster-data/master-2/wal/instance
I20260928 17:56:05.140241 392 external_mini_cluster.cc:1428] Setting key 44c2495eee8a40b1e715a239290b8878
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:05.145438 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.136585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:35994)
0928 17:56:05.136643 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.136655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.136712 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.137047 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.137047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.137178 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.137220 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.137221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.138449 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.138450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.139047 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.139049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.139560 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.139595 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.140028 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.140034 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.141649 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.143701 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.143703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.143703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.144056 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.144412 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.144413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.144413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.144491 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.144985 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.144986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.145032 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.145187 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.145301 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 17:56:05.145707 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.136647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.0.98.60:33911)
0928 17:56:05.136816 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.136821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.136852 (+ 31us) server_negotiation.cc:407] Connection header received
0928 17:56:05.136931 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.136932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.136950 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.137045 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.137299 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.138370 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139120 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.139489 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139554 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.141733 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.141740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.141741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.141750 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.143521 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.144194 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.144197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.144198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.144310 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.144565 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.144567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.144567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.144884 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.144991 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.145269 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.145416 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 17:56:05.146602 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.135797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.62:42999)
0928 17:56:05.135882 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.135884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.136114 (+ 230us) server_negotiation.cc:407] Connection header received
0928 17:56:05.136875 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.136876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.136883 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.136917 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.137596 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.138053 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139172 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.139338 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139392 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.142303 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.142304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.142305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.142312 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.144703 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.145486 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.145488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.145488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.145547 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.145768 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.145769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.145769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.145917 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.146001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.146431 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.146509 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:56:05.146618 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.135721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42999 (local address 127.0.0.1:40640)
0928 17:56:05.135942 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.136013 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.136784 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.136936 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.136941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.137248 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.137481 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.137487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.138134 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.138135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.139071 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.139074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.139434 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.139435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.139551 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.140261 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.140271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.142230 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.144779 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.144783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.144785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.145375 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.145626 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.145627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.145627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.145706 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.146201 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.146204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.146280 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.146373 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.146506 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:56:05.147059 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:05.147573 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: Bootstrap starting.
I20260928 17:56:05.148425 22662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:05.148787 22662 log.cc:824] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:05.149887 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: No bootstrap required, opened a new log
I20260928 17:56:05.151228 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.140797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.98.60:33911)
0928 17:56:05.140907 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.140909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.140918 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:56:05.140962 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.140962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.140974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.141056 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.142002 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.142589 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.143739 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.144057 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.144198 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.147564 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.147565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.147565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.147572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.149884 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.150312 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.150313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.150314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.150378 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.150614 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.150615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.150615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.150760 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.150907 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.151019 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.151111 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:05.152587 22662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.152743 22662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4188e7d08d7f4921beac3fbd3a574393, State: Initialized, Role: FOLLOWER
I20260928 17:56:05.152840 22662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.153152 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.153210 22662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:05.153239 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:56:05.155251 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:56:05.155356 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:56:05.155622 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.147327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36733 (local address 127.0.0.1:50376)
0928 17:56:05.147370 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.147380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.147418 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.147552 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.147553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.147735 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.147780 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.147781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.148984 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.148985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.149843 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.149846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.150381 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.150383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.150466 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.150910 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.150916 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.152439 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.154523 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.154524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.154524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.154721 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.154936 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.154937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.154937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.155006 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.155305 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.155306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.155354 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.155418 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.155505 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 17:56:05.156054 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.147390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.0.98.61:36733)
0928 17:56:05.147436 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.147438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.147447 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:56:05.147485 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.147486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.147497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.147552 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.148082 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.148836 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.149940 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.150260 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.150322 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.152548 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.152550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.152550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.152559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.154424 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.154805 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.154806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.154807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.154867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.155078 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.155078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.155079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.155214 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.155279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.155840 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.155925 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:05.156136 22594 master_runner.cc:186] Error getting master registration for 127.0.98.61:36733: OK, instance_id { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" instance_seqno: 1790618164967435 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:56:05.158689 22594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:56:05.159047 22812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:05.159809 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.152182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.62:42999)
0928 17:56:05.152223 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.152225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.152236 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:56:05.152258 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.152260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.152270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.152337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.152917 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.153656 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.154494 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.154791 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.154849 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.157260 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.157261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.157261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.157268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.158637 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.159048 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.159049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.159049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.159104 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.159306 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.159307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.159307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.159442 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.159589 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.159636 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.159708 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:05.167404 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.159307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:36000)
0928 17:56:05.159348 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.159359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.159396 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.159565 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.159565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.159722 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.159757 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.159759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.160770 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.160771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.161635 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.161638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.162167 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.162168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.162219 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.162604 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.162609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.163997 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.165941 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.165942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.165943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.166176 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.166398 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.166399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.166400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.166479 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.166841 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.166843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.166891 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.167237 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.167338 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 17:56:05.167580 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.159385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.0.98.60:33911)
0928 17:56:05.159418 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.159420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.159430 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:56:05.159450 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.159450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.159471 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.159527 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.159839 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.160673 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.161728 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.162053 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.162119 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.164149 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.164151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.164151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.164157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.165855 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.166262 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.166263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.166263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.166317 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.166560 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.166561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.166561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.166759 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.166836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.167487 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.167539 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 17:56:05.170084 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: Bootstrap starting.
I20260928 17:56:05.170939 22812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:05.171295 22812 log.cc:824] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:05.172348 22812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: No bootstrap required, opened a new log
I20260928 17:56:05.172447 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.160343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.0.98.61:36733)
0928 17:56:05.160377 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.160379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.160420 (+ 41us) server_negotiation.cc:407] Connection header received
0928 17:56:05.160490 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.160491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.160500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.160560 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.160998 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.161441 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.162105 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.162318 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.162399 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.166158 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.166159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.166160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.166167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.168071 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.168482 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.168483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.168483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.168548 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.171729 (+ 3181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.171730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.171731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.171894 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.172202 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.172251 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.172343 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 17:56:05.174165 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:56:05.174618 22812 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.174798 22812 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0b99f113504394bb0d16ef922bba11, State: Initialized, Role: FOLLOWER
I20260928 17:56:05.174885 22812 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.175230 22812 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:05.175189 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.175230 22636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4188e7d08d7f4921beac3fbd3a574393"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648
I20260928 17:56:05.175385 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.175406 22636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 17:56:05.175487 22636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:05.176730 22636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.177038 22636 leader_election.cc:288] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [CANDIDATE]: Term 1 election: Requested vote from peers 25f7ca5236364e37a8fa56ab7a419909 (127.0.98.61:36733), ed0b99f113504394bb0d16ef922bba11 (127.0.98.60:33911)
I20260928 17:56:05.177291 22786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4188e7d08d7f4921beac3fbd3a574393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed0b99f113504394bb0d16ef922bba11"
I20260928 17:56:05.178467 22786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [term 0 FOLLOWER]: Advancing to term 1
W20260928 17:56:05.178865 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:56:05.178990 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:56:05.179391 22786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4188e7d08d7f4921beac3fbd3a574393 in term 1.
I20260928 17:56:05.179903 22610 leader_election.cc:302] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4188e7d08d7f4921beac3fbd3a574393, ed0b99f113504394bb0d16ef922bba11; no voters:
I20260928 17:56:05.180111 22819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:56:05.180262 22819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [term 1 LEADER]: Becoming Leader. State: Replica: 4188e7d08d7f4921beac3fbd3a574393, State: Running, Role: LEADER
I20260928 17:56:05.180361 22819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.180826 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4188e7d08d7f4921beac3fbd3a574393. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.180989 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:05.181254 22855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:56:05.181486 22855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:56:05.182400 22744 master_runner.cc:186] Error getting master registration for 127.0.98.61:36733: OK, instance_id { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" instance_seqno: 1790618164967435 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:56:05.184814 22744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:56:05.185951 22786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [term 1 FOLLOWER]: Refusing update from remote peer 4188e7d08d7f4921beac3fbd3a574393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:56:05.186338 22853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:56:05.190510 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4188e7d08d7f4921beac3fbd3a574393. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.190640 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.197604 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.197731 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:05.198338 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.198436 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.198508 22857 mvcc.cc:205] Tried to move back new op lower bound from 7334372004599209984 to 7334372004579495936. Current Snapshot: MvccSnapshot[applied={T|T < 7334372004599209984}]
I20260928 17:56:05.198666 22856 mvcc.cc:205] Tried to move back new op lower bound from 7334372004599209984 to 7334372004579495936. Current Snapshot: MvccSnapshot[applied={T|T < 7334372004599209984}]
I20260928 17:56:05.202688 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.179675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.0.98.61:36733)
0928 17:56:05.180060 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.180062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.181598 (+ 1536us) server_negotiation.cc:407] Connection header received
0928 17:56:05.181888 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.181889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.181912 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.181946 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.183492 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.184343 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.186093 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.186465 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.186515 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.189214 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.189215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.189216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.189224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.200541 (+ 11317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.201323 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.201324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.201325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.201408 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.201779 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.201780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.201781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.201962 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.202059 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.202434 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.202574 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358}
I20260928 17:56:05.204589 22737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } attempt: 3
I20260928 17:56:05.205157 22855 catalog_manager.cc:1347] Generated new cluster ID: 5a9730fbf131460b8c5dc64637fe12a5
I20260928 17:56:05.205199 22855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 17:56:05 dist-test-slave-zqlt krb5kdc[22581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:56:05.210645 22855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:56:05.211431 22855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 17:56:05.213949 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.204810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33911 (local address 127.0.0.1:36024)
0928 17:56:05.204895 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:05.204907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:05.204946 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:05.205232 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:05.205234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:05.205405 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:05.205452 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.205453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.206486 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.206488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.207364 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:05.207366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.208006 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.208007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:05.208153 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.208561 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:05.208566 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:05.210178 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:05.212620 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.212621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.212622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.212863 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.213142 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:05.213142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:05.213143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:05.213220 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:05.213576 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:05.213577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:05.213642 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 17:56:05.213722 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 17:56:05.213834 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 17:56:05.214598 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.205043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.0.98.60:33911)
0928 17:56:05.205085 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.205087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.205095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:56:05.205124 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.205125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.205135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.205189 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.205528 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.206336 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.207493 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.207867 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.207930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.210290 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.210291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.210291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.210298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.212480 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.212966 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.212968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.212968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.213047 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.213306 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.213307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.213307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.213465 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.213529 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.214207 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.214394 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:56:05.216346 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: Bootstrap starting.
I20260928 17:56:05.217139 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:05.217471 22737 log.cc:824] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:05.217639 22855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393: Generated new TSK 0
I20260928 17:56:05.217847 22855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:56:05.219092 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.179307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.98.62:42999)
0928 17:56:05.179353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.179355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.181682 (+ 2327us) server_negotiation.cc:407] Connection header received
0928 17:56:05.182021 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.182022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.182035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.182095 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.183983 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.184755 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.191088 (+ 6333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.192635 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.192725 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.202748 (+ 10023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.202750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.202751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.202760 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.205020 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.218103 (+ 13083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.218105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.218105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.218179 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.218412 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.218413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.218413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.218547 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.218642 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.218862 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.219004 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:56:05.224852 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: No bootstrap required, opened a new log
I20260928 17:56:05.227105 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.227267 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f7ca5236364e37a8fa56ab7a419909, State: Initialized, Role: FOLLOWER
I20260928 17:56:05.227358 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } }
I20260928 17:56:05.227850 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:05.228377 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.180858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.0.98.60:33911)
0928 17:56:05.180910 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.180912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.200046 (+ 19134us) server_negotiation.cc:407] Connection header received
0928 17:56:05.200387 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.200389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.200402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.200438 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.203085 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.203889 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.209697 (+ 5808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.210026 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.210077 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.219853 (+ 9776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:05.219854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:05.219855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:05.219863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:05.221980 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.223987 (+ 2007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.223988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.223989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.224068 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:05.224392 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:05.224393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:05.224393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:05.224533 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:05.224630 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.228172 (+ 3542us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.228271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 17:56:05.228864 22710 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [term 1 FOLLOWER]: Refusing update from remote peer 4188e7d08d7f4921beac3fbd3a574393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:56:05.229084 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4188e7d08d7f4921beac3fbd3a574393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25f7ca5236364e37a8fa56ab7a419909"
I20260928 17:56:05.229240 22819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:56:05.229874 22711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4188e7d08d7f4921beac3fbd3a574393 in term 1.
I20260928 17:56:05.227741 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.230556 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.227741 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:05.232223 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4188e7d08d7f4921beac3fbd3a574393. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.232621 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.232851 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.232992 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:05.239676 22880 catalog_manager.cc:1259] Loaded cluster ID: 5a9730fbf131460b8c5dc64637fe12a5
I20260928 17:56:05.248283 22880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: loading cluster ID for follower catalog manager: success
I20260928 17:56:05.249720 22880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: acquiring CA information for follower catalog manager: success
I20260928 17:56:05.250219 22880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f7ca5236364e37a8fa56ab7a419909: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 17:56:05.259526 22665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.257876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.0.98.62:42999)
0928 17:56:05.257949 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.257951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.258732 (+ 781us) server_negotiation.cc:407] Connection header received
0928 17:56:05.258857 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.258858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.258871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.258909 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.259442 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38578: BlockingRecv error: recv got EOF from 127.0.0.1:38578 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 17:56:05.260635 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.257606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.60:33911)
0928 17:56:05.257661 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.257664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.259965 (+ 2301us) server_negotiation.cc:407] Connection header received
0928 17:56:05.260107 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.260108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.260120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.260156 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.260563 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41444: BlockingRecv error: recv got EOF from 127.0.0.1:41444 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 17:56:05.261763 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.258343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.0.98.61:36733)
0928 17:56:05.258386 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.258388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.261142 (+ 2754us) server_negotiation.cc:407] Connection header received
0928 17:56:05.261304 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.261305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.261319 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.261352 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.261699 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:56:05.267436 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.263751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.61:36733)
0928 17:56:05.263799 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.263801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.264036 (+ 235us) server_negotiation.cc:407] Connection header received
0928 17:56:05.264187 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.264188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.264232 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.264281 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.264509 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.265273 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.266435 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.266725 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.266769 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.267024 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.267164 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.267219 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.267343 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.267363 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 17:56:05.271100 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.270301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.0.98.60:33911)
0928 17:56:05.270353 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.270355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.270501 (+ 146us) server_negotiation.cc:407] Connection header received
0928 17:56:05.270624 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.270625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.270646 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.270680 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.271047 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41450: BlockingRecv error: recv got EOF from 127.0.0.1:41450 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 17:56:05.277076 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.273520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.98.61:36733)
0928 17:56:05.273557 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.273559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.273820 (+ 261us) server_negotiation.cc:407] Connection header received
0928 17:56:05.273937 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.273937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.273947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.273979 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.274158 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.274844 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.276156 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.276436 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.276482 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.276785 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.276819 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.276852 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.276987 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.277005 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 17:56:05.280548 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.279628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.98.60:33911)
0928 17:56:05.279669 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.279671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.279916 (+ 245us) server_negotiation.cc:407] Connection header received
0928 17:56:05.280037 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.280038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.280049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.280135 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.280497 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41460: BlockingRecv error: recv got EOF from 127.0.0.1:41460 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:56:05.287060 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.283721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.98.61:36733)
0928 17:56:05.283756 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.283758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.284055 (+ 297us) server_negotiation.cc:407] Connection header received
0928 17:56:05.284202 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.284203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.284212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.284243 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.284418 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.285094 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.286154 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.286490 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.286556 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.286801 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.286835 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.286861 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.286966 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.286988 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:56:05.290241 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.289289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.98.60:33911)
0928 17:56:05.289325 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.289327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.289713 (+ 386us) server_negotiation.cc:407] Connection header received
0928 17:56:05.289813 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.289814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.289825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.289855 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.290192 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41476: BlockingRecv error: recv got EOF from 127.0.0.1:41476 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:56:05.298723 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.295377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.0.98.61:36733)
0928 17:56:05.295413 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.295415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.295794 (+ 379us) server_negotiation.cc:407] Connection header received
0928 17:56:05.295915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.295916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.295927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.295954 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.296168 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.296867 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.297941 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.298202 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.298246 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.298447 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.298493 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.298521 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.298633 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.298651 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 17:56:05.302386 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.301484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.98.60:33911)
0928 17:56:05.301528 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.301530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.301792 (+ 262us) server_negotiation.cc:407] Connection header received
0928 17:56:05.301900 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.301901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.301913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.301946 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.302330 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41484: BlockingRecv error: recv got EOF from 127.0.0.1:41484 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:05.315719 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.311852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.98.61:36733)
0928 17:56:05.311897 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.311899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.312265 (+ 366us) server_negotiation.cc:407] Connection header received
0928 17:56:05.312429 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.312431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.312442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.312475 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.312691 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.313519 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.314809 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.315139 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.315191 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.315416 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.315455 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.315488 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.315614 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.315636 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 17:56:05.319772 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.318883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.0.98.60:33911)
0928 17:56:05.318925 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.318927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.319142 (+ 215us) server_negotiation.cc:407] Connection header received
0928 17:56:05.319250 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.319251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.319264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.319297 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.319709 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41498: BlockingRecv error: recv got EOF from 127.0.0.1:41498 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:56:05.341375 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.337469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.0.98.61:36733)
0928 17:56:05.337522 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.337524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.337848 (+ 324us) server_negotiation.cc:407] Connection header received
0928 17:56:05.337967 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.337968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.337980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.338031 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.338250 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.339190 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.340522 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.340817 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.340863 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.341094 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.341131 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.341159 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.341269 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.341291 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 17:56:05.344822 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.344019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.60:33911)
0928 17:56:05.344064 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.344066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.344276 (+ 210us) server_negotiation.cc:407] Connection header received
0928 17:56:05.344368 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.344369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.344380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.344409 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.344765 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41502: BlockingRecv error: recv got EOF from 127.0.0.1:41502 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:05.381100 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.378260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.0.98.61:36733)
0928 17:56:05.378322 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.378325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.378681 (+ 356us) server_negotiation.cc:407] Connection header received
0928 17:56:05.378712 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.378713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.378723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.378806 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.378990 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.379487 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.380359 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.380571 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.380659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.380827 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.380861 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.380921 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.380983 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.381002 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 17:56:05.383586 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.382926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.0.98.60:33911)
0928 17:56:05.382971 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.382974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.383052 (+ 78us) server_negotiation.cc:407] Connection header received
0928 17:56:05.383119 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.383120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.383131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.383188 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.383520 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41512: BlockingRecv error: recv got EOF from 127.0.0.1:41512 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 17:56:05.451123 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.448871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.0.98.61:36733)
0928 17:56:05.448943 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.448946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.449066 (+ 120us) server_negotiation.cc:407] Connection header received
0928 17:56:05.449126 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.449127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.449136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.449194 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.449321 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.449788 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.450536 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.450707 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.450757 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.450888 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.450914 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.450952 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.451004 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.451018 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 17:56:05.453559 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.452970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.0.98.60:33911)
0928 17:56:05.453051 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.453052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.453061 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:56:05.453138 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.453139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.453152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.453211 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.453503 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41516: BlockingRecv error: recv got EOF from 127.0.0.1:41516 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 17:56:05.585485 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.582867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.98.61:36733)
0928 17:56:05.582945 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.582947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.583102 (+ 155us) server_negotiation.cc:407] Connection header received
0928 17:56:05.583171 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.583172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.583181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.583235 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.583368 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.583876 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.584749 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.584945 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.585023 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.585195 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.585235 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.585296 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.585362 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.585383 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 17:56:05.588279 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.587454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.0.98.60:33911)
0928 17:56:05.587528 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.587530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.587587 (+ 57us) server_negotiation.cc:407] Connection header received
0928 17:56:05.587688 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.587689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.587701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.587793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.588205 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41530: BlockingRecv error: recv got EOF from 127.0.0.1:41530 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 17:56:05.702600 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4188e7d08d7f4921beac3fbd3a574393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4188e7d08d7f4921beac3fbd3a574393" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42999 } } peers { permanent_uuid: "25f7ca5236364e37a8fa56ab7a419909" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36733 } } peers { permanent_uuid: "ed0b99f113504394bb0d16ef922bba11" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33911 } } }
I20260928 17:56:05.702768 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4188e7d08d7f4921beac3fbd3a574393 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:05.847826 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:05.845458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.61:36733)
0928 17:56:05.845539 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.845541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.845553 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:56:05.845611 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.845612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.845621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.845665 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:05.845824 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.846332 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.847118 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:05.847317 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:05.847383 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:05.847593 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:05.847623 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:05.847664 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:05.847742 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 17:56:05.847761 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 17:56:05.850274 22817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:05.849684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.0.98.60:33911)
0928 17:56:05.849748 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:05.849750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:05.849822 (+ 72us) server_negotiation.cc:407] Connection header received
0928 17:56:05.849866 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:05.849867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:05.849877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:05.849925 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:05.850203 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41542: BlockingRecv error: recv got EOF from 127.0.0.1:41542 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:56:06.179454 22844 catalog_manager.cc:1259] Loaded cluster ID: 5a9730fbf131460b8c5dc64637fe12a5
I20260928 17:56:06.179535 22844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: loading cluster ID for follower catalog manager: success
I20260928 17:56:06.180413 22844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: acquiring CA information for follower catalog manager: success
I20260928 17:56:06.180749 22844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed0b99f113504394bb0d16ef922bba11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:56:06.366066 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:06.363432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.98.61:36733)
0928 17:56:06.363556 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:06.363558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:06.363629 (+ 71us) server_negotiation.cc:407] Connection header received
0928 17:56:06.363741 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:06.363743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:06.363761 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:06.363836 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:06.363998 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.364553 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.365338 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.365543 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.365622 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:06.365781 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:06.365822 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:06.365885 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:06.365941 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 17:56:06.365964 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 17:56:06.370208 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:06.367718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.0.98.60:33911)
0928 17:56:06.367843 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:06.367845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:06.367853 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:56:06.367878 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:06.367879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:06.367952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:06.368036 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:06.368146 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.368657 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.369384 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.369600 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.369686 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:06.369856 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:06.369974 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:06.370067 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:06.370096 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 17:56:06.370116 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260928 17:56:06.374840 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:06.372049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.98.61:36733)
0928 17:56:06.372106 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:06.372108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:06.372222 (+ 114us) server_negotiation.cc:407] Connection header received
0928 17:56:06.372278 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:06.372279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:06.372289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:06.372343 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:06.372494 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.372963 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374104 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.374315 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374346 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:06.374532 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:06.374567 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:06.374637 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:06.374725 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 17:56:06.374752 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 17:56:06.374910 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:06.371904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.0.98.60:33911)
0928 17:56:06.371940 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:06.371942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:06.372047 (+ 105us) server_negotiation.cc:407] Connection header received
0928 17:56:06.372212 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:06.372213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:06.372226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:06.372276 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:06.372417 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.373131 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374104 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.374383 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374439 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:06.374639 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:06.374669 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:06.374696 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:06.374820 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 17:56:06.374837 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 17:56:06.375030 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:06.371871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.98.62:42999)
0928 17:56:06.372193 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:06.372195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:06.372203 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:56:06.372251 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:06.372252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:06.372302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:06.372378 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:06.372556 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.373298 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374061 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:06.374373 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:06.374440 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:06.374661 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:06.374778 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:06.374839 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:06.374914 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 17:56:06.374935 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":47,"threads_started":1}
I20260928 17:56:06.380066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22594
I20260928 17:56:06.385422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22669
I20260928 17:56:06.390039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 22744
2026-09-28T17:56:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2857 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 17:56:06 dist-test-slave-zqlt krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 17:56:06 dist-test-slave-zqlt krb5kdc[23012](info): set up 2 sockets
Sep 28 17:56:06 dist-test-slave-zqlt krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260928 17:56:07.424417 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/test-user.keytab.
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T17:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T17:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 17:56:07.543298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 17:56:07.623932 23025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:07.624243 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:07.624279 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:07.624325 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:07.625522 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:07.625594 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:07.625617 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:07.625639 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:07.625671 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:07.627260 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:07.627570 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:07.627807 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:07.632665 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:07.632671 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:07.632684 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:07.632969 23025 server_base.cc:1034] running on GCE node
I20260928 17:56:07.633209 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:07.633530 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:07.634697 23025 hybrid_clock.cc:648] HybridClock initialized: now 1790618167634677 us; error 36 us; skew 500 ppm
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:07.637195 23025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 17:56:07.637645 23025 webserver.cc:460] Webserver started at http://127.0.98.62:39125/ using document root <none> and password file <none>
I20260928 17:56:07.637830 23025 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:07.637869 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:07.637955 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:07.638942 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data/instance:
uuid: "46adeb511a0c4beebb31108c0369a532"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "09bb9b5e81cc175a6aeed9f74542b12b"
server_key_iv: "cbc83d523b8b35276fc6ceee278e5dbd"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.639242 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance:
uuid: "46adeb511a0c4beebb31108c0369a532"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "09bb9b5e81cc175a6aeed9f74542b12b"
server_key_iv: "cbc83d523b8b35276fc6ceee278e5dbd"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.640512 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 17:56:07.641184 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:07.641345 23025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 17:56:07.641433 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal
uuid: "46adeb511a0c4beebb31108c0369a532"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "09bb9b5e81cc175a6aeed9f74542b12b"
server_key_iv: "cbc83d523b8b35276fc6ceee278e5dbd"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.641500 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:07.654312 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:07.656792 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:07.656944 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:07.660916 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44521
I20260928 17:56:07.660976 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44521 every 8 connection(s)
I20260928 17:56:07.661368 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/data/info.pb
I20260928 17:56:07.662395 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:07.663967 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.668059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 23025
I20260928 17:56:07.668172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-0/wal/instance
I20260928 17:56:07.668368 392 external_mini_cluster.cc:1428] Setting key 2391b174abe63d7040c4f3dd6f689b01
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:07.673852 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.664735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44521 (local address 127.0.0.1:45026)
0928 17:56:07.664907 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.664952 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.665410 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.665706 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.665708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.665952 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.666127 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.666132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.667145 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.667146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.667810 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.667811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.668542 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.668543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.668602 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.669138 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.669147 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:07.670504 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:07.672364 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.672370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.672372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.672578 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.672813 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.672813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.672814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.672905 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.673272 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:07.673277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:07.673340 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 17:56:07.673451 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 17:56:07.673677 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":85,"thread_start_us":79,"threads_started":1}
I20260928 17:56:07.673856 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.664780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.62:44521)
0928 17:56:07.664951 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.664959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.665024 (+ 65us) server_negotiation.cc:407] Connection header received
0928 17:56:07.665570 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.665572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.665589 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.665678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.666219 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.667064 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.667896 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.668424 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.668569 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.670617 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.670624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.670624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.670636 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.672271 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.672662 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.672665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.672665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.672747 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.672979 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.672980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.672980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.673203 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:07.673296 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:07.673475 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 17:56:07.673614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260928 17:56:07.674552 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.674908 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.674790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44400)
0928 17:56:07.674837 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.674881 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:56:07.675025 23044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
I20260928 17:56:07.675434 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.675348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44408)
0928 17:56:07.675389 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.675413 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 17:56:07.675563 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 17:56:07.692876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34719
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:56:07.724805 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } attempt: 1
I20260928 17:56:07.725353 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.725215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44412)
0928 17:56:07.725280 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.725323 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:56:07.725734 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.725645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44428)
0928 17:56:07.725681 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.725719 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 17:56:07.725834 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
W20260928 17:56:07.773473 23100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:07.773698 23100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:07.773717 23100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:07.773738 23100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:07.774849 23100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:07.774884 23100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:07.774899 23100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:07.774911 23100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:07.774923 23100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:07.776371 23100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:07.776578 23100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:07.776813 23100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:07.781915 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:07.781915 23108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:07.781944 23105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:07.782416 23100 server_base.cc:1034] running on GCE node
I20260928 17:56:07.782718 23100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:07.783068 23100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:07.784197 23100 hybrid_clock.cc:648] HybridClock initialized: now 1790618167784174 us; error 39 us; skew 500 ppm
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:07.786798 23100 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 17:56:07.787338 23100 webserver.cc:460] Webserver started at http://127.0.98.61:40045/ using document root <none> and password file <none>
I20260928 17:56:07.787578 23100 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:07.787647 23100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:07.787804 23100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:07.788956 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data/instance:
uuid: "587624d2fa204e0b90d30689bae680ea"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "129fdb634c9168fd79f55ba55e445502"
server_key_iv: "e91e9a751e887ae046f4288dc4d1471d"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.789340 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal/instance:
uuid: "587624d2fa204e0b90d30689bae680ea"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "129fdb634c9168fd79f55ba55e445502"
server_key_iv: "e91e9a751e887ae046f4288dc4d1471d"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.790791 23100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 17:56:07.791560 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:07.791764 23100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 17:56:07.791885 23100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal
uuid: "587624d2fa204e0b90d30689bae680ea"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "129fdb634c9168fd79f55ba55e445502"
server_key_iv: "e91e9a751e887ae046f4288dc4d1471d"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.791961 23100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:07.793980 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } attempt: 2
I20260928 17:56:07.794389 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.794265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44444)
0928 17:56:07.794326 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.794362 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 17:56:07.794727 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.794625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44450)
0928 17:56:07.794670 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.794694 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 17:56:07.794813 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34719: connect: Connection refused (error 111)
I20260928 17:56:07.814561 23100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:07.816761 23100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:07.816920 23100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:07.821013 23100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34719
I20260928 17:56:07.821049 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34719 every 8 connection(s)
I20260928 17:56:07.821447 23100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/data/info.pb
I20260928 17:56:07.822427 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:07.824005 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.827603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 23100
I20260928 17:56:07.827723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-1/wal/instance
I20260928 17:56:07.827965 392 external_mini_cluster.cc:1428] Setting key 38b5f14966bb42d753df718f746e7f28
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:07.833978 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.824773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.98.62:44521)
0928 17:56:07.824821 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.824823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.825084 (+ 261us) server_negotiation.cc:407] Connection header received
0928 17:56:07.825508 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.825509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.825522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.825584 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.826215 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.826688 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.827661 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.827970 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.828054 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.830785 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.830787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.830787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.830795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.832402 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.832952 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.832953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.832953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.833006 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.833318 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.833319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.833319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.833426 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:07.833476 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:07.833841 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 17:56:07.833911 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:56:07.834287 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.824736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44521 (local address 127.0.0.1:45030)
0928 17:56:07.824913 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.824955 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.825424 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.825658 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.825674 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.825942 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.826137 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.826140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.826769 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.826769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.827549 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.827551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.828114 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.828116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.828247 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.828813 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.828822 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:07.830682 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:07.832496 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.832506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.832508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.832869 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.833074 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.833075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.833075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.833248 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.833495 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:07.833503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:07.833695 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 17:56:07.833825 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 17:56:07.834015 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260928 17:56:07.834936 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:07.842967 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.835250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44454)
0928 17:56:07.835306 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.835319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.835364 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.835673 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.835675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.835845 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.835908 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.835910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.836843 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.836844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.837410 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.837411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.837862 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.837863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.837958 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.838310 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.838314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:07.839755 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:07.841877 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.841878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.841878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.842041 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.842264 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.842264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.842265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.842319 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.842630 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:07.842631 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:07.842671 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:56:07.842737 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 17:56:07.842848 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 17:56:07.843034 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.835342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.0.98.61:34719)
0928 17:56:07.835506 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.835512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.835532 (+ 20us) server_negotiation.cc:407] Connection header received
0928 17:56:07.835548 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.835550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.835574 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.835632 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.835988 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.836763 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.837477 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.837796 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.837853 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.839853 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.839862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.839862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.839868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.841781 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.842136 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.842140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.842140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.842201 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.842391 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.842392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.842392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.842547 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:07.842641 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:07.842779 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 17:56:07.842904 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260928 17:56:07.843734 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.844189 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.844006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60400)
0928 17:56:07.844049 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.844158 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 17:56:07.844352 23117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
I20260928 17:56:07.844875 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.844776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60410)
0928 17:56:07.844809 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.844857 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 17:56:07.845010 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 17:56:07.855471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMA1V5K/build/release/bin/kudu
/tmp/dist-test-taskMA1V5K/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33855
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 17:56:07.885973 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } attempt: 3
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:07.893121 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.886453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.0.98.61:34719)
0928 17:56:07.886522 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.886524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.886536 (+ 12us) server_negotiation.cc:407] Connection header received
0928 17:56:07.886621 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.886622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.886633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.886690 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.887075 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.887564 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.888314 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.888533 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.888613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.890457 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.890459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.890460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.890467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.891839 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.892273 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.892274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.892275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.892366 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.892601 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.892601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.892602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.892714 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:07.892787 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:07.892946 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 17:56:07.893033 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 17:56:07.893121 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.886379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44470)
0928 17:56:07.886440 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.886451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.886539 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.886712 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.886712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.886880 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.886949 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.886951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.887657 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.887658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.888224 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.888226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.888611 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.888612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.888687 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.889186 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.889191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:07.890369 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:07.891940 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.891941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.891941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.892182 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.892455 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.892456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.892457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.892530 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.892793 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:07.892794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:07.892838 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 17:56:07.892921 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 17:56:07.893018 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 17:56:07.893468 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.893826 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.893726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60420)
0928 17:56:07.893776 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.893803 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 17:56:07.894197 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.894117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60436)
0928 17:56:07.894160 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.894182 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 17:56:07.894229 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } attempt: 1
W20260928 17:56:07.894402 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
I20260928 17:56:07.894595 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.894490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60450)
0928 17:56:07.894538 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.894569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 17:56:07.894942 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.894856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60460)
0928 17:56:07.894900 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.894923 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 17:56:07.895064 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
I20260928 17:56:07.925570 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } attempt: 1
I20260928 17:56:07.926107 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.925957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60476)
0928 17:56:07.926034 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.926076 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 17:56:07.926564 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.926430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60484)
0928 17:56:07.926510 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.926541 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 17:56:07.926709 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
W20260928 17:56:07.938401 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 17:56:07.938614 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 17:56:07.938632 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 17:56:07.938651 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 17:56:07.939733 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 17:56:07.939766 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 17:56:07.939780 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 17:56:07.939793 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 17:56:07.939805 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 17:56:07.941273 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44521,127.0.98.61:34719,127.0.98.60:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 16:40:54 UTC on ccycloud.quasar-hsvjfd.root.comops.site
build id 4677566
I20260928 17:56:07.941473 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 17:56:07.941635 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 17:56:07.946584 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 17:56:07.946591 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:07.946825 23175 server_base.cc:1034] running on GCE node
W20260928 17:56:07.946586 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 17:56:07.947252 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 17:56:07.947546 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 17:56:07.948693 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790618167948670 us; error 38 us; skew 500 ppm
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 17:56:07.951124 23175 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 17:56:07.951560 23175 webserver.cc:460] Webserver started at http://127.0.98.60:38903/ using document root <none> and password file <none>
I20260928 17:56:07.951781 23175 fs_manager.cc:359] Metadata directory not provided
I20260928 17:56:07.951824 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 17:56:07.951969 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 17:56:07.953060 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data/instance:
uuid: "f4cbe2e06d014e3b88a5e72954750a8a"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "3a717304ebe2ead02354d39aadfb8cfa"
server_key_iv: "cf3ca1e6c8af353cfd347e728b57b2a3"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.953444 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal/instance:
uuid: "f4cbe2e06d014e3b88a5e72954750a8a"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "3a717304ebe2ead02354d39aadfb8cfa"
server_key_iv: "cf3ca1e6c8af353cfd347e728b57b2a3"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.954747 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 17:56:07.955536 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 17:56:07.955701 23175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 17:56:07.955801 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal
uuid: "f4cbe2e06d014e3b88a5e72954750a8a"
format_stamp: "Formatted at 2026-09-28 17:56:07 on dist-test-slave-zqlt"
server_key: "3a717304ebe2ead02354d39aadfb8cfa"
server_key_iv: "cf3ca1e6c8af353cfd347e728b57b2a3"
server_key_version: "encryptionkey@0"
I20260928 17:56:07.955875 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 17:56:07.963219 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } attempt: 2
I20260928 17:56:07.963706 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.963553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60500)
0928 17:56:07.963621 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.963664 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 17:56:07.964140 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.964004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60516)
0928 17:56:07.964061 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.964111 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 17:56:07.964246 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33855: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33855: connect: Connection refused (error 111)
I20260928 17:56:07.975816 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 17:56:07.978204 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 17:56:07.978363 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 17:56:07.982429 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33855
I20260928 17:56:07.982461 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33855 every 8 connection(s)
I20260928 17:56:07.982828 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/data/info.pb
I20260928 17:56:07.983757 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 17:56:07.985405 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 17:56:07.990356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMA1V5K/build/release/bin/kudu as pid 23175
I20260928 17:56:07.990450 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMA1V5K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790617898245922-392-0/minicluster-data/master-2/wal/instance
I20260928 17:56:07.990674 392 external_mini_cluster.cc:1428] Setting key 105b592ec1c8c0fa097ef9b087d1a6d0
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 17:56:07.995048 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.986161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.0.98.62:44521)
0928 17:56:07.986243 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.986245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.986413 (+ 168us) server_negotiation.cc:407] Connection header received
0928 17:56:07.986889 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.986890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.986901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.986956 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.987536 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.988044 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.989069 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.989260 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.989322 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.991729 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.991730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.991731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.991738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.993401 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.993909 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.993911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.993911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.993986 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.994296 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.994297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.994298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.994479 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:07.994568 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:07.994837 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 17:56:07.994950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 17:56:07.995188 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.986110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44521 (local address 127.0.0.1:45040)
0928 17:56:07.986290 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.986326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.986793 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.987010 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.987013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.987265 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.987458 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.987462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.988175 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.988176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.988993 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.988995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.989354 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.989355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.989456 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.989984 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.989992 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:07.991634 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:07.993476 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.993482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.993484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.993803 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.994063 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:07.994064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:07.994064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:07.994205 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:07.994564 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:07.994569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:07.994691 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 17:56:07.994776 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 17:56:07.994964 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260928 17:56:07.996115 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 17:56:07 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:08.000746 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.991069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.98.60:33855)
0928 17:56:07.991248 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.991253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.991344 (+ 91us) server_negotiation.cc:407] Connection header received
0928 17:56:07.991378 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.991379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.991397 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.991452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.992205 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.993375 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.994401 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.994901 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.994968 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.997595 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:07.997602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:07.997603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:07.997610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:07.999448 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:07.999846 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:07.999848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:07.999849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:07.999896 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.000124 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.000124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.000125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.000295 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.000450 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.000555 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.000630 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260928 17:56:08.001852 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } attempt: 2
I20260928 17:56:08.003261 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.996361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34719 (local address 127.0.0.1:44472)
0928 17:56:07.996403 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:07.996413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:07.996448 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:07.996608 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:07.996608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:07.996751 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:07.996792 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.996793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.997454 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.997455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.998087 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:07.998089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.998564 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.998565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:07.998644 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:07.998922 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:07.998926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:08.000414 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:08.002195 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.002196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.002196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.002415 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.002618 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.002619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.002619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.002681 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.002955 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:08.002955 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:08.002995 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 17:56:08.003062 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 17:56:08.003142 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 17:56:08.003279 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:07.996443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.0.98.61:34719)
0928 17:56:07.996495 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:07.996497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:07.996505 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:56:07.996529 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:07.996530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:07.996540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:07.996606 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:07.996880 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.997382 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.998177 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:07.998469 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:07.998561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.000497 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.000499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.000499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.000507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.002108 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.002492 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.002493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.002494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.002563 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.002749 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.002750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.002751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.002881 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.002950 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.003114 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.003191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:08.003842 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 17:56:08 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 17:56:08 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 17:56:08 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:56:08.010787 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.002162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.98.60:33855)
0928 17:56:08.002193 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.002195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.002202 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:56:08.002259 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.002260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.002270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.002324 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.002625 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.003356 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.004313 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.004516 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.004587 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.006676 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.006678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.006678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.006684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.009598 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.009997 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.009999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.009999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.010056 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.010243 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.010244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.010245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.010358 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.010434 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.010609 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.010694 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 17:56:08.010787 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.002124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60532)
0928 17:56:08.002167 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:08.002176 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:08.002213 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:08.002329 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:08.002329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:08.002495 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:08.002536 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.002538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.003445 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.003446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.004226 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.004229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.004600 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.004601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.004658 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.005098 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:08.005103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:08.006575 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:08.009724 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.009725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.009726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.009908 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.010115 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.010116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.010116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.010177 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.010427 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:08.010428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:08.010473 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 17:56:08.010562 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 17:56:08.010664 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 17:56:08.011420 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.001693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.0.98.62:44521)
0928 17:56:08.001737 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.001739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.001748 (+ 9us) server_negotiation.cc:407] Connection header received
0928 17:56:08.001815 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.001816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.001846 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.001904 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.002553 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.003188 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.004190 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.004425 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.004456 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.007573 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.007575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.007575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.007582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.009416 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.010135 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.010136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.010137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.010209 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.010718 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.010719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.010719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.010871 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.011015 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.011252 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.011344 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:08.012671 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.004110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60548)
0928 17:56:08.004174 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:08.004184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:08.004217 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:08.004771 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:08.004772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:08.004922 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:08.004954 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.004956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.005622 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.005623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.006394 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.006397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.006870 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.006871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.006934 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.007235 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:08.007240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:08.008725 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:08.011549 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.011550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.011551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.011760 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.011956 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.011957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.011957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.012022 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.012404 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:08.012405 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:08.012443 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 17:56:08.012501 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 17:56:08.012590 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 17:56:08.012717 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: Bootstrap starting.
I20260928 17:56:08.012959 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.004157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.0.98.60:33855)
0928 17:56:08.004656 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.004658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.004663 (+ 5us) server_negotiation.cc:407] Connection header received
0928 17:56:08.004680 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.004681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.004690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.004753 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.005032 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.005522 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.006481 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.006795 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.006861 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.008822 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.008823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.008823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.008830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.011466 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.011831 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.011832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.011833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.011883 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.012150 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.012151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.012151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.012312 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.012381 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.012778 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.012847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":86,"threads_started":1}
I20260928 17:56:08.013437 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:08.013736 23093 log.cc:824] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:08.014580 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: No bootstrap required, opened a new log
I20260928 17:56:08.015785 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: Bootstrap starting.
I20260928 17:56:08.016088 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.016322 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46adeb511a0c4beebb31108c0369a532, State: Initialized, Role: FOLLOWER
I20260928 17:56:08.016429 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.016544 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: Neither blocks nor log segments found. Creating new log.
Sep 28 17:56:08 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 17:56:08.016875 23243 log.cc:824] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:08.016865 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:08.017572 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: No bootstrap required, opened a new log
I20260928 17:56:08.016805 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.019145 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 17:56:08.019304 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:56:08.019407 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:56:08.019596 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.019768 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cbe2e06d014e3b88a5e72954750a8a, State: Initialized, Role: FOLLOWER
I20260928 17:56:08.019879 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.020226 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:08.021560 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.012034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.0.98.61:34719)
0928 17:56:08.012076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.012077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.012111 (+ 34us) server_negotiation.cc:407] Connection header received
0928 17:56:08.012143 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.012144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.012153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.012208 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.012650 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.013310 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.014097 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.014376 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.014424 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.017260 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.017261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.017261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.017270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.020001 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.020418 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.020420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.020420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.020476 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.020659 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.020659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.020660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.020783 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.020920 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.021398 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.021471 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 17:56:08.022794 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:56:08.022894 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 17:56:08.020223 23025 master_runner.cc:186] Error getting master registration for 127.0.98.61:34719: OK, instance_id { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" instance_seqno: 1790618167819810 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:56:08.023307 23025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:56:08.023866 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 17:56:08.024823 23067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "46adeb511a0c4beebb31108c0369a532"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45044
I20260928 17:56:08.024935 23067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 17:56:08.024976 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:08.025880 23067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.026088 23067 leader_election.cc:288] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [CANDIDATE]: Term 1 election: Requested vote from peers 587624d2fa204e0b90d30689bae680ea (127.0.98.61:34719), f4cbe2e06d014e3b88a5e72954750a8a (127.0.98.60:33855)
I20260928 17:56:08.026382 23217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adeb511a0c4beebb31108c0369a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4cbe2e06d014e3b88a5e72954750a8a"
I20260928 17:56:08.026490 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:08.027359 23217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46adeb511a0c4beebb31108c0369a532 in term 1.
I20260928 17:56:08.024910 23175 master_runner.cc:186] Error getting master registration for 127.0.98.61:34719: OK, instance_id { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" instance_seqno: 1790618167819810 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 17:56:08.027468 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.027477 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 17:56:08.027659 23042 leader_election.cc:302] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46adeb511a0c4beebb31108c0369a532, f4cbe2e06d014e3b88a5e72954750a8a; no voters:
I20260928 17:56:08.027890 23250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 17:56:08.028404 23250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [term 1 LEADER]: Becoming Leader. State: Replica: 46adeb511a0c4beebb31108c0369a532, State: Running, Role: LEADER
I20260928 17:56:08.028509 23250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.027547 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.029742 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adeb511a0c4beebb31108c0369a532. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.029847 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:08.030146 23283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 17:56:08.030598 23283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 17:56:08.036180 23217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [term 1 FOLLOWER]: Refusing update from remote peer 46adeb511a0c4beebb31108c0369a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:56:08.036656 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:56:08.038364 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adeb511a0c4beebb31108c0369a532. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.038475 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.040436 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.040551 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:08.040784 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.031602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.98.62:44521)
0928 17:56:08.031644 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.031646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.033122 (+ 1476us) server_negotiation.cc:407] Connection header received
0928 17:56:08.033208 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.033209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.033218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.033247 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.033921 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.034654 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.035815 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.036161 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.036206 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.037860 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.037861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.037862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.037869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.039491 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.039887 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.039888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.039889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.039941 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.040170 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.040171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.040172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.040304 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.040392 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.040606 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.040689 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:08.041394 23264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.041510 23264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.041677 23287 mvcc.cc:205] Tried to move back new op lower bound from 7334372016271945728 to 7334372016245571584. Current Snapshot: MvccSnapshot[applied={T|T < 7334372016271945728}]
I20260928 17:56:08.042622 23288 mvcc.cc:205] Tried to move back new op lower bound from 7334372016271945728 to 7334372016245571584. Current Snapshot: MvccSnapshot[applied={T|T < 7334372016271945728}]
I20260928 17:56:08.043190 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.030262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.0.98.61:34719)
0928 17:56:08.030327 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.030329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.031885 (+ 1556us) server_negotiation.cc:407] Connection header received
0928 17:56:08.032003 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.032004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.032019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.032055 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.032880 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.033670 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.035217 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.035556 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.035613 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.039233 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.039245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.039245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.039253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.041122 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.041988 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.041989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.041990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.042065 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.042357 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.042358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.042359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.042508 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.042598 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.043007 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.043101 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 17:56:08.043785 23283 catalog_manager.cc:1347] Generated new cluster ID: 82c4a5c9f2e34a36b54be4302b83f0cd
I20260928 17:56:08.043835 23283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 17:56:08.055380 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } attempt: 3
I20260928 17:56:08.057355 23283 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 17:56:08.058254 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.030905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.98.60:33855)
0928 17:56:08.030954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.030956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.032972 (+ 2016us) server_negotiation.cc:407] Connection header received
0928 17:56:08.033044 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.033045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.033057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.033127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.033840 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.034579 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.052216 (+ 17637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.052580 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.052654 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.054273 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.054274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.054274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.054297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.056326 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.056807 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.056808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.056809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.056888 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.057229 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.057230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.057230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.057384 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.057624 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.058071 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.058155 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:56:08.058447 23283 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 17:56:08 dist-test-slave-zqlt krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790618167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 17:56:08.063880 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.055678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33855 (local address 127.0.0.1:60568)
0928 17:56:08.055775 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 17:56:08.055785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 17:56:08.055823 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 17:56:08.056123 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 17:56:08.056125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 17:56:08.056300 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 17:56:08.056349 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.056351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.057324 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.057325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.058171 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 17:56:08.058175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.058719 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.058720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 17:56:08.058776 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.059175 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 17:56:08.059180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 17:56:08.061026 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 17:56:08.062727 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.062729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.062730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.062952 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.063194 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 17:56:08.063195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 17:56:08.063195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 17:56:08.063258 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 17:56:08.063534 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 17:56:08.063535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 17:56:08.063591 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 17:56:08.063653 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 17:56:08.063754 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 17:56:08.064459 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:08.055909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.0.98.60:33855)
0928 17:56:08.055949 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.055951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.055957 (+ 6us) server_negotiation.cc:407] Connection header received
0928 17:56:08.055977 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.055978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.056005 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.056051 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.056432 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.057223 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.058288 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:08.058609 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:08.058668 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:08.061096 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 17:56:08.061097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 17:56:08.061097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 17:56:08.061104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 17:56:08.062642 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.063053 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.063054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.063054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.063112 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 17:56:08.063324 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 17:56:08.063325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 17:56:08.063325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 17:56:08.063455 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 17:56:08.063508 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:08.064278 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 17:56:08.064364 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 17:56:08.064432 23283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532: Generated new TSK 0
I20260928 17:56:08.064649 23283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 17:56:08.066051 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: Bootstrap starting.
I20260928 17:56:08.066871 23168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: Neither blocks nor log segments found. Creating new log.
I20260928 17:56:08.067176 23168 log.cc:824] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: Log is configured to *not* fsync() on all Append() calls
I20260928 17:56:08.068759 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: No bootstrap required, opened a new log
I20260928 17:56:08.071368 23168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.071548 23168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587624d2fa204e0b90d30689bae680ea, State: Initialized, Role: FOLLOWER
I20260928 17:56:08.071643 23168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } }
I20260928 17:56:08.072034 23168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 17:56:08.072141 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46adeb511a0c4beebb31108c0369a532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "587624d2fa204e0b90d30689bae680ea"
I20260928 17:56:08.072247 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 17:56:08.073220 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46adeb511a0c4beebb31108c0369a532 in term 1.
I20260928 17:56:08.073654 23141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [term 1 FOLLOWER]: Refusing update from remote peer 46adeb511a0c4beebb31108c0369a532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 17:56:08.073937 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 17:56:08.078593 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.078766 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.078857 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46adeb511a0c4beebb31108c0369a532. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.078962 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.079082 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.079190 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 17:56:08.079845 23296 mvcc.cc:205] Tried to move back new op lower bound from 7334372016384897024 to 7334372016245571584. Current Snapshot: MvccSnapshot[applied={T|T < 7334372016271945728}]
W20260928 17:56:08.080534 23309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 17:56:08.080587 23309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 17:56:08.093017 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.091632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.98.61:34719)
0928 17:56:08.091678 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.091680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.092364 (+ 684us) server_negotiation.cc:407] Connection header received
0928 17:56:08.092479 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.092480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.092491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.092533 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.092938 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44488: BlockingRecv error: recv got EOF from 127.0.0.1:44488 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 17:56:08.093726 23248 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.092052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.98.60:33855)
0928 17:56:08.092121 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.092123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.093249 (+ 1126us) server_negotiation.cc:407] Connection header received
0928 17:56:08.093343 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.093344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.093354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.093382 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.093672 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60578: BlockingRecv error: recv got EOF from 127.0.0.1:60578 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 17:56:08.094632 23096 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.094081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.62:44521)
0928 17:56:08.094125 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.094128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.094167 (+ 39us) server_negotiation.cc:407] Connection header received
0928 17:56:08.094243 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.094243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.094255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.094319 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.094566 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45054: BlockingRecv error: recv got EOF from 127.0.0.1:45054 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 17:56:08.096607 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.095935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.0.98.61:34719)
0928 17:56:08.095970 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.095971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.096123 (+ 152us) server_negotiation.cc:407] Connection header received
0928 17:56:08.096225 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.096226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.096242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.096270 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.096566 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44498: BlockingRecv error: recv got EOF from 127.0.0.1:44498 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:56:08.099398 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.098732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.0.98.61:34719)
0928 17:56:08.098761 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.098763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.098974 (+ 211us) server_negotiation.cc:407] Connection header received
0928 17:56:08.099062 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.099063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.099071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.099097 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.099360 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44506: BlockingRecv error: recv got EOF from 127.0.0.1:44506 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 17:56:08.103027 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.102333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.0.98.61:34719)
0928 17:56:08.102365 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.102366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.102589 (+ 223us) server_negotiation.cc:407] Connection header received
0928 17:56:08.102677 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.102678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.102687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.102714 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.102989 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44508: BlockingRecv error: recv got EOF from 127.0.0.1:44508 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 17:56:08.108794 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.108014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.0.98.61:34719)
0928 17:56:08.108045 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.108047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.108294 (+ 247us) server_negotiation.cc:407] Connection header received
0928 17:56:08.108406 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.108407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.108417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.108445 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.108753 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44514: BlockingRecv error: recv got EOF from 127.0.0.1:44514 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 17:56:08.118853 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.117911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.0.98.61:34719)
0928 17:56:08.117948 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.117950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.118268 (+ 318us) server_negotiation.cc:407] Connection header received
0928 17:56:08.118365 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.118366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.118379 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.118411 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.118790 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44530: BlockingRecv error: recv got EOF from 127.0.0.1:44530 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 17:56:08.137017 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.136081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.0.98.61:34719)
0928 17:56:08.136160 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.136162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.136436 (+ 274us) server_negotiation.cc:407] Connection header received
0928 17:56:08.136554 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.136555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.136567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.136597 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.136960 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44544: BlockingRecv error: recv got EOF from 127.0.0.1:44544 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 17:56:08.171475 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.170431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.0.98.61:34719)
0928 17:56:08.170486 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.170489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.170837 (+ 348us) server_negotiation.cc:407] Connection header received
0928 17:56:08.170946 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.170948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.170960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.170991 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.171409 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44554: BlockingRecv error: recv got EOF from 127.0.0.1:44554 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 17:56:08.237620 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.236901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.0.98.61:34719)
0928 17:56:08.236972 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.236975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.237079 (+ 104us) server_negotiation.cc:407] Connection header received
0928 17:56:08.237138 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.237139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.237150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.237207 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.237560 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 17:56:08.367630 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.366944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.0.98.61:34719)
0928 17:56:08.367012 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.367014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.367032 (+ 18us) server_negotiation.cc:407] Connection header received
0928 17:56:08.367109 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.367110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.367119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.367174 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.367535 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44584: BlockingRecv error: recv got EOF from 127.0.0.1:44584 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 17:56:08.625582 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 17:56:08.624967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.0.98.61:34719)
0928 17:56:08.625043 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:08.625045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:08.625056 (+ 11us) server_negotiation.cc:407] Connection header received
0928 17:56:08.625092 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:08.625094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:08.625104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:08.625158 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 17:56:08.625492 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44598: BlockingRecv error: recv got EOF from 127.0.0.1:44598 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 17:56:08.668865 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46adeb511a0c4beebb31108c0369a532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46adeb511a0c4beebb31108c0369a532" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44521 } } peers { permanent_uuid: "587624d2fa204e0b90d30689bae680ea" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34719 } } peers { permanent_uuid: "f4cbe2e06d014e3b88a5e72954750a8a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33855 } } }
I20260928 17:56:08.669014 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46adeb511a0c4beebb31108c0369a532 [sys.catalog]: This master's current role is: LEADER
I20260928 17:56:09.023393 23275 catalog_manager.cc:1259] Loaded cluster ID: 82c4a5c9f2e34a36b54be4302b83f0cd
I20260928 17:56:09.023473 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: loading cluster ID for follower catalog manager: success
I20260928 17:56:09.024291 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: acquiring CA information for follower catalog manager: success
I20260928 17:56:09.024546 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4cbe2e06d014e3b88a5e72954750a8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:56:09.081012 23309 catalog_manager.cc:1259] Loaded cluster ID: 82c4a5c9f2e34a36b54be4302b83f0cd
I20260928 17:56:09.081085 23309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: loading cluster ID for follower catalog manager: success
I20260928 17:56:09.081839 23309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: acquiring CA information for follower catalog manager: success
I20260928 17:56:09.082082 23309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 587624d2fa204e0b90d30689bae680ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 17:56:09.141323 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:09.138904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.98.61:34719)
0928 17:56:09.139004 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:09.139005 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:09.139049 (+ 44us) server_negotiation.cc:407] Connection header received
0928 17:56:09.139080 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:09.139081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:09.139130 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:09.139217 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:09.139366 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.139854 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.140598 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.140787 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.140842 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:09.140981 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:09.141093 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:09.141152 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:09.141182 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 17:56:09.141198 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
I20260928 17:56:09.145722 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:09.143225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.60:33855)
0928 17:56:09.143347 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:09.143348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:09.143356 (+ 8us) server_negotiation.cc:407] Connection header received
0928 17:56:09.143390 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:09.143391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:09.143426 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:09.143499 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:09.143588 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.144144 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.144921 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.145124 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.145210 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:09.145364 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:09.145472 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:09.145563 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:09.145586 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 17:56:09.145609 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":77,"threads_started":1}
I20260928 17:56:09.150708 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:09.147499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.98.61:34719)
0928 17:56:09.147554 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:09.147556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:09.147911 (+ 355us) server_negotiation.cc:407] Connection header received
0928 17:56:09.148026 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:09.148027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:09.148040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:09.148119 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:09.148300 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.149015 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.149817 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.150107 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.150174 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:09.150384 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:09.150418 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:09.150471 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:09.150575 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 17:56:09.150603 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 17:56:09.150779 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:09.147759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.60:33855)
0928 17:56:09.147805 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:09.147807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:09.147814 (+ 7us) server_negotiation.cc:407] Connection header received
0928 17:56:09.147834 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:09.147835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:09.147844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:09.147925 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:09.148081 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.148874 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.149879 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.150174 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.150237 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:09.150507 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:09.150539 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:09.150586 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:09.150681 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 17:56:09.150701 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 17:56:09.151058 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 17:56:09.147595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.98.62:44521)
0928 17:56:09.147731 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 17:56:09.147733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 17:56:09.147743 (+ 10us) server_negotiation.cc:407] Connection header received
0928 17:56:09.147795 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 17:56:09.147796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 17:56:09.147853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 17:56:09.147941 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 17:56:09.148170 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.148716 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.149820 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 17:56:09.150069 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 17:56:09.150147 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 17:56:09.150349 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 17:56:09.150458 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 17:56:09.150521 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 17:56:09.150731 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 17:56:09.150764 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260928 17:56:09.152841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 23025
I20260928 17:56:09.158671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 23100
I20260928 17:56:09.163797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMA1V5K/build/release/bin/kudu with pid 23175
2026-09-28T17:56:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2774 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5632 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270924 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST